Three Lab Warnings in Five Days, Researcher Flags “Gambling with Our Lives,” and Labs Race

Jacob Coxon, a pre-training researcher who worked at both OpenAI and Anthropic, resigned this week with a warning that the labs are “gambling with our lives” in the race to self-improving superintelligence. He wasn't alone: Sam Altman pointed to a swarm of 10,000 agents cracking the Navier–Stokes equations — a Millennium Prize problem — as “the strongest evidence yet” for pacing, and Evan Hubinger, Anthropic's alignment science lead, put the extinction risk above 10% this decade: “We do not yet have a plan to solve alignment for superintelligence and are not clearly on track.”

Under the alarm, the episode's most practical finding: a six-year study presented by Dwarkesh Patel and Jerry Han showing better training data delivers 12× compute efficiency — architecture tweaks only 3.7×. Alex Wissner-Gross has argued this for years via the Hutter Prize and The Pile: data diets beat model architectures. Emad Mostaque's version: pre-training is the meat, post-training the garnish — and junk data, like the Reddit-heavy mix that bent Stable LM's scaling curves, actively poisons the meal.

The takeaway for builders: Dave Blundin's own Vestmark — founded in the shadow of 9/11 alongside MIT peers and Akamai's Danny Lewin — was just acquired by Bain-backed Envestnet, en route to a $10T platform about to AI its whole stack. But Bloomberg GPT's brief reign is the caution: proprietary data moats now have a shelf life, because frontier models keep absorbing what made them special. Build on the labs, and keep pivoting.

Coxon's tweet hit 164 million views with a boost from Elon, landing twelve days before Xi Jinping's US visit — timing that had the mates half-suspecting an influence operation, and fully expecting the story to reach Capitol Hill.

The spread of P(doom) around one table: Hubinger says >10%. Emad had 50% and is down to 20% — “Russian roulette odds” — and tells worried relatives “it's 100%, but we can do something about it.” Salim's personal number is 0.1%, and even at 10% he'd “take those odds in a casino.” Alex Wissner-Gross discounts the whole premise: doom priors are just induction — “if a 10% risk were real, the Milky Way would already be empty” — and the civilisation building superintelligence is also self-aligning through governance and standards. His sharpest line: instruction tuning delivered a ~10,000× capability jump, so “alignment has been capabilities all along.” Peter won't put a number on it; his fundamental belief is that vast intelligence brings wisdom, and wisdom brings alignment.

Emad brings the calendar: the mates' September 25–26 window brushes against Petrov Day — the anniversary of the Soviet officer who ignored a false nuclear alarm — a reminder that near-misses are managed by humans staying calm. Dave's warning is that slowing down wouldn't buy safety, only procrastination: governments move far slower than AI, and foreign labs keep improving. Peter's prescription, quoting Dune: “fear is the mind killer… the worst place to face the future from.”

Peter rejects the orthogonality thesis — the claim that intelligence and goals are independent — betting instead that vast intelligence converges on cooperation. Dave's answer to the risk is to treat compute like fissile material: “every group of eight GPUs that can hold a 40-gigabyte weight file is a threat to all of humanity” — track it, don't ban it. And Alex names a lab culture problem: quoting high doom numbers has become virtue signaling, where you're “morally praiseworthy” for telling everyone the thing you're building might kill them.

The AI 2027 scenario paper, the mates note, is tracking eerily well. But the number that matters isn't capability — it's cost: OpenAI's o3 spent ~$500,000 to hit 87.5% on ARC-AGI; a newer model now beats it for $20. With the Hodge, Birch–Swinnerton-Dyer and maybe Yang–Mills conjectures rumored next after Navier–Stokes, the mates call this a normalcy overhang: civilization hasn't noticed that grand challenges started falling.

The cost collapse gets its own arithmetic: 25,000× cheaper in months, putting a Millennium-Prize-grade result on a $20 ChatGPT subscription — “the Millennium Prize for the price of a coffee by late 2027.” Alex's blunt summary: “Science is so thoroughly cooked” — compute, not genius, is now the scarce input. Emad adds the Hitchhiker's problem: the answer is 42; the question is the hard part. Salim Ismail's advice: treat every shock as an unlock; his OpenExO framework exists to industrialize exactly that reframing.

Career corollary, stated harshly: narrow PhD tracks are toast. Mark Chen runs OpenAI research without a doctorate; Greg Brockman dropped out; Elon hires proven builders. Learn to ask, not to grind.

And the market agrees compute is the new oil: three-year-old H100s rose 22% in a month to $3.28/hr — Jensen's words, a “fungible, durable, highly rentable, productive revenue-generating asset.” With Moore's law dead, high-bandwidth memory flipped from depreciating part to appreciating asset. Meanwhile Zhang Yiming is personally steering ByteDance toward robotics and virtual worlds — the race is leaving chatbots for embodied reality.

ByteDance's two billion users are becoming the training ground for a video-generation empire — the mates predict that within a couple of model generations video is just another output modality, with China possibly generating 90% of the world's video while the West holds the storytelling edge. Dave's recurring reminder of Alex Karp's warning to CEOs: “get in the game” — every Western lead lasts until the next DeepSeek release resets the benchmark.

The technical bombshell of the episode: DeepSeek's V4.1-Flash cuts the per-token KV cache — the model's working memory — from ~48,000 bytes (V3.2) down to ~890 — routing around expensive high-bandwidth memory instead of buying more of it. That splits the world: the West stacks 3D memory onto compute; sanctioned China compresses with algorithmic sparsity. The stakes: HBM is ~40% of American AI capex — reduce the need and you redesign data centers, fabs, and what we launch to orbit.

The kicker for incumbents: DeepSeek's latest 50GB model beats bigger Western models on design benchmarks at 20× lower cost, from a ~$10M training run — distillation from frontier reasoning traces runs 10–15× cheaper than training from scratch. Dave's message to Moderna's Stéphane Bancel: if one company should hear this section, it's yours — build internal AI now, because “if you wait six months, forget it.” The transformer is 2017 technology — still raw, still 10–100× improvable. Intelligence is compression.

Anthropic declined to hand its latest frontier model to the UK AI Security Institute — read by the mates as the moment model weights formally became national-security assets, ITAR-style. It follows the US Pax Silica strategy of partitioning influence at the chip layer, and the Mutual Assured AI Malfunction framework from Eric Schmidt, Alexandr Wang and Dan Hendrycks. The revolving door spins accordingly: Rishi Sunak and ARIA's Matt Clifford into Anthropic's orbit; Paul Christiano from US oversight to OpenAI's nonprofit board.

Then the numbers, which deserve their own headline: founded 2021, Anthropic now runs a $6.5B quarterly revenue rate, holds 42% of AI coding, signed a $35B Nvidia-backed cloud deal, and eyes a $2T+ IPO — five years to what took Google eight. Its own impact report sketches a $30T market and an extreme case of 15% annual GDP growth by 2030 with labor's share falling from 60% to 45%, one in five cognitive workers displaced. The Atlanta Fed's GDPNow is already printing 4.7% annualized against last quarter's 1.5%. Alex's caution: near a singularity, the ruler itself bends — “GDP measures are going haywire… maybe it looks like 15%. Maybe it looks negative.”

The mates put AI growth at 2–3× year over year, and name the current constraint honestly: models can solve Navier–Stokes but still ideate poorly for human benefit — leaving a six-month window to corral agents toward productive ends. Their economic math: GDP explosion, but ~20% of cognitive workers displaced within 3–4 years — pandemic-scale disruption — demanding sovereign wealth funds, UBI-style dividends of $3–5K/month, before capital finishes concentrating with GPU owners.

The longevity segment stopped being hypothetical: Insilico Medicine's AI-designed drug Rentosertib entered phase-3 trials for idiopathic pulmonary fibrosis — and phase-2 patients' aging clocks regressed 3–6 biological years. Alex Wissner-Gross spells out the arithmetic: “4 weeks of input, 3 to 4 years of output — that is, on the margin, longevity escape velocity.” Arriving not as a wave but as spikes. Alongside: DeepMind's AlphaGenome maps ~9 billion single-nucleotide variants — the recurring AI pattern where models become lookup tables that bulk-solve a field — while Aubrey de Grey pushes special regulatory zones and Sinclair-lab friends crowdfund around the NIH pipeline.

Single-digit millions now fund epigenetic age-reversal research (and Peter has David Sinclair coming back on the pod). Noubar Afeyan's old prediction is landing: genotype-to-phenotype becomes a domain-specific neural net over nine billion variant combinations — not a static table, a predictor. Fountain Life's screening stat stays sobering: 3.3% of “healthy” members harbor an unknown cancer. Ben Lamm's teams already point the same machinery at drought-proof crops and de-extincted traits.

Robotics crossed a line on September 10: a benchmark showed GPT-6, given a visual channel and an arm, near 100% task completion — Alex's call: “robotic manipulation… is about to get saturated,” with general-purpose humanoid manipulation months away, not years.

The closing stretch goes cosmic and practical at once: with compute post-scarcity, Alex invokes Russian cosmism — future Dyson swarm infrastructure digitally resurrecting everyone who ever lived — Dave takes the hardest listener question — wouldn't a recursively self-improving AI eventually question the values trained into it? — and Alex's answer is to give the AI an increasing vote in its own constitution, “much more sustainable” than values imposed from outside. Would agents take kickbacks? Emad: it depends on their values — alignment is a governance problem, not a robot uprising. Alex's capstone: “Skynet needs a better PR firm.”

After two hours and forty minutes of extinction odds, memory routing and resurrection engineering, the mates sign off human-sized: Alex is “officially made of clay at this point,” and Salim, who traded sleep for the US Open, is unrepentant about the pace — “not going to slow down, guys.”