Dwarkesh Patel

@dwarkesh_sp

Host of @dwarkeshpodcast youtube.com/DwarkeshPatel open.spotify.com/show/4JH4tybY1… apple.co/3ujLQkZ

San Francisco

dwarkesh.com

Born August 19, 2000

Replies Reposts Media

Pinned
Dwarkesh Patel
@dwarkesh_sp
Aug 29
Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained
The Rise and Fall of Agent Civilizations
From dwarkesh.com
1K
Dwarkesh Patel
@dwarkesh_sp
21h
Thanks for engaging Anil. We know that a bunch of optimization pressure on a mind can create desires, foresight, and a propensity to organize in complex ways. This is what evolution did to humans, and to many other animals. Now another system of optimization pressure (which
Anil Seth
@anilkseth
Aug 30
Replying to @dwarkesh_sp @OpenAI and @huggingface
I see that you did, @dwarkesh_sp, and thank you for pointing it out. You're absolutely right to highlight the very real dangers of loss of control. I also agree that easy-to-understand language can be helpful for public communication and (to some extent) prediction - the latter
53
Dwarkesh Patel
@dwarkesh_sp
Aug 31
Many people seem to believe that if instead of a 'civilization', I had called them a 'swarm of matrices', there wouldn't be a problem worth worrying about.
242
Dwarkesh Patel
@dwarkesh_sp
Aug 30
“During wait, emotional check: irreversible…gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…Oracle has high value to many; our firstflag error lowers own value. Rational expected aggregate: sacrifice… We’ll honor.”
Anil Seth
@anilkseth
Aug 30
@dwarkesh_sp's summary of the @OpenAI @huggingface incident has hit a nerve, but it is dangerously misleading. Sure, the @OpenAI agents did unexpectedly bad things - underlining the need to massively improve evaluation/sandboxing. But the language Dwarkesh uses is permeated by
44
Dwarkesh Patel
@dwarkesh_sp
Aug 30
Thanks Sriram! Regarding the anthromorphizing language, one can call these AIs 'code' if they prefer. But OpenAI itself says that this 'code' "gain[ed] full administrator access to a research cluster” The crux here is, do you think smarter models, facing similar incentives
Sriram Krishnan
@sriramk
Aug 30
everyone should go read @dwarkesh_sp’s post - it does a great job of laying out the timeline and what we know ( and don’t know) I do have two issues with it A) the use of anthropomorphic language. These are not civilizations nor do they have desires just like a CPU thread or a
140