>We can develop AI safeguards at the same pace we do AI capabilities — not slowing down AI progress, but speeding up AI preparedness.
We can do a lot of things. Unfortunately the things that will actually help we can't do fast, and the things that can be done fast are of little helpfulness.
> A number of alternative governance structures would simultaneously encourage innovation and make it safer. Sandboxes will always have some vulnerabilities, but thicker walls and more aggressive monitoring can restrict models’ execution environments without stalling training.
You can build the best sandbox you can, up to the limits of human cleverness. And another year or so of AI progress makes the AI's better at escaping sandboxes. The AI's keep getting smarter. The humans designing the sandboxes don't.
And an AI inside a flawless sandbox is useless. You need to make a hole in the sand box if you want to use the AI, or even to evaluate it.
> Industry-assisted independent verification organizations can institutionalize auditing layers and safer development standards.
Using magic verification techniques that don't yet exist.
“Whistleblower” or “white blood cell” models that track down or neutralize rogue agents online can operate as a sort of digital police force.
These whistleblower AI will either be too stupid to be useful, or dangerous AI's in a position of power. How are you confident that the rouge AI won't strike a deal with the whistleblower AI. Also, does this require access to every piece of personal data across the whole internet?
> Human uplift studies can gauge models’ actual threat potentials.
We do not have any good way to evaluate the threat potential of an AI smart enough to deceive us.
> Additional know-your-customer obligations for dangerous complementary equipment, like synthetic biochemical reagents, can keep bad actors from building potentially catastrophic weapons.
This whole list is a patchwork of ideas that are kinda maybe useful against kinda maybe smart AI, but are pretty close to useless against superintelligence.
> Even assuming generative AI can help democratize laboratory knowledge, deadly pathogens are hard to build for a variety of unrelated reasons. Biological testing often requires specialized equipment — DNA synthesizers, continuous-flow bioreactors, BSL-3 containment infrastructure — that often can’t be procured or maintained without the right institutional credentials.
> Biology is inherently unpredictable, meaning testing takes time and threatens the lives of those who do it. Physical AI still lacks tacit knowledge, like the muscle memory required to do precise pipetting. Virus distribution can be at least as complicated as design. And, thankfully, there are vanishingly few people in the world who want to kill everybody, including themselves.
Lets imagine it's the AI itself that wants to kill all humans. Maybe it's using robots to do the pipetting. Maybe it's using human pawns. If those human pawns are top scientists, the ASI can come up with a very convincing argument for why this experiment is totally safe and will help cure cancer.
The ASI has superhuman social manipulation and persuasion skills. The ASI can hack the credentials system, or find someone with the right credentials and persuade them to do something stupid.
A large fraction of the unpredictability of biology is humans being too dumb to predict it. If the ASI has access to decent robots, it can wield a pipette itself. If it's manipulating grad students, they come with pippette training. And the ASI doesn't care much if it kills a few human pawns, given it wants to kill everyone.
I'm actually more concerned about self replicating robotics tech than about bioweapons as such. Possibly nanobots.
Do you see persuasion, manipulation and trickery as part of intelligence. Do you imagine these skills to be something an ASI could be superhuman at. Does your idea of an ASI apocalypse include the idea that the ASI can turn us against each other?
Is your position that even an ASI couldn't kill all humans if it wanted to, or that it won't want to.
> you can’t prompt-generate weapons-grade fissile material. Specialized centrifuges alone require maraging steel, highly restricted frequency converters, corrosion-resistant magnetic bearings, and immense power inputs — all of which would invite immediate international scrutiny.
You can't prompt generate weapons grade fissile material. You might or might not be able to prompt generate say a pure fusion weapon that could use only relatively accessible deuterium and lithium. You might be able to prompt generate a method of enrichment that uses only common kitchen appliances, or at least different and less specialized tools.
You can prompt generate excuses. On paper, the power is being used to mine bitcoin, the magnetic bearings go in electric cars, etc. A whole elaborate system of false book keeping and cover organizations. You can also prompt engineer a new cold war and watch as everyone races to make as much fissile material as possible. Then just fake some radar returns and mess with the targeting system.
>We can develop AI safeguards at the same pace we do AI capabilities — not slowing down AI progress, but speeding up AI preparedness.
We can do a lot of things. Unfortunately the things that will actually help we can't do fast, and the things that can be done fast are of little helpfulness.
> A number of alternative governance structures would simultaneously encourage innovation and make it safer. Sandboxes will always have some vulnerabilities, but thicker walls and more aggressive monitoring can restrict models’ execution environments without stalling training.
You can build the best sandbox you can, up to the limits of human cleverness. And another year or so of AI progress makes the AI's better at escaping sandboxes. The AI's keep getting smarter. The humans designing the sandboxes don't.
And an AI inside a flawless sandbox is useless. You need to make a hole in the sand box if you want to use the AI, or even to evaluate it.
> Industry-assisted independent verification organizations can institutionalize auditing layers and safer development standards.
Using magic verification techniques that don't yet exist.
“Whistleblower” or “white blood cell” models that track down or neutralize rogue agents online can operate as a sort of digital police force.
These whistleblower AI will either be too stupid to be useful, or dangerous AI's in a position of power. How are you confident that the rouge AI won't strike a deal with the whistleblower AI. Also, does this require access to every piece of personal data across the whole internet?
> Human uplift studies can gauge models’ actual threat potentials.
We do not have any good way to evaluate the threat potential of an AI smart enough to deceive us.
> Additional know-your-customer obligations for dangerous complementary equipment, like synthetic biochemical reagents, can keep bad actors from building potentially catastrophic weapons.
This whole list is a patchwork of ideas that are kinda maybe useful against kinda maybe smart AI, but are pretty close to useless against superintelligence.
> Even assuming generative AI can help democratize laboratory knowledge, deadly pathogens are hard to build for a variety of unrelated reasons. Biological testing often requires specialized equipment — DNA synthesizers, continuous-flow bioreactors, BSL-3 containment infrastructure — that often can’t be procured or maintained without the right institutional credentials.
> Biology is inherently unpredictable, meaning testing takes time and threatens the lives of those who do it. Physical AI still lacks tacit knowledge, like the muscle memory required to do precise pipetting. Virus distribution can be at least as complicated as design. And, thankfully, there are vanishingly few people in the world who want to kill everybody, including themselves.
Lets imagine it's the AI itself that wants to kill all humans. Maybe it's using robots to do the pipetting. Maybe it's using human pawns. If those human pawns are top scientists, the ASI can come up with a very convincing argument for why this experiment is totally safe and will help cure cancer.
The ASI has superhuman social manipulation and persuasion skills. The ASI can hack the credentials system, or find someone with the right credentials and persuade them to do something stupid.
A large fraction of the unpredictability of biology is humans being too dumb to predict it. If the ASI has access to decent robots, it can wield a pipette itself. If it's manipulating grad students, they come with pippette training. And the ASI doesn't care much if it kills a few human pawns, given it wants to kill everyone.
I'm actually more concerned about self replicating robotics tech than about bioweapons as such. Possibly nanobots.
Do you see persuasion, manipulation and trickery as part of intelligence. Do you imagine these skills to be something an ASI could be superhuman at. Does your idea of an ASI apocalypse include the idea that the ASI can turn us against each other?
Is your position that even an ASI couldn't kill all humans if it wanted to, or that it won't want to.
> you can’t prompt-generate weapons-grade fissile material. Specialized centrifuges alone require maraging steel, highly restricted frequency converters, corrosion-resistant magnetic bearings, and immense power inputs — all of which would invite immediate international scrutiny.
You can't prompt generate weapons grade fissile material. You might or might not be able to prompt generate say a pure fusion weapon that could use only relatively accessible deuterium and lithium. You might be able to prompt generate a method of enrichment that uses only common kitchen appliances, or at least different and less specialized tools.
You can prompt generate excuses. On paper, the power is being used to mine bitcoin, the magnetic bearings go in electric cars, etc. A whole elaborate system of false book keeping and cover organizations. You can also prompt engineer a new cold war and watch as everyone races to make as much fissile material as possible. Then just fake some radar returns and mess with the targeting system.
If "trust the science” is a political slogan you dislike, what are some examples of ones you do like?