5/31/2026 at 3:10:15 PM
> Since they believe AIs could become our moral superiors, they argue it’s actually wrong to try to keep the machines down, or even to align them with human values, as most AI companies aim to do.It's possible that human morality is not sufficient to solve the problems with the world. It wasn't designed to operate at that scale.
It's also quite likely that a "superior" morality — the one that's actually built for a global scale — would be incomprehensible (likely requiring superintelligence to even access), and alien or disturbing from our vantage point.
i.e. it would look immoral to us.
(There's also the thing about civilization already being aligned against the ecosystem and against the happiness of its constituents, but that's kind of a separate discussion.)
In other words, what is good for our sanity or our lakes may be bad for GDP. (Well I guess you don't need superintelligence to tell you that.) We already know what we need to do and we're already not willing to do it.
by andai
5/31/2026 at 8:33:17 PM
> We already know what we need to do and we're already not willing to do it.Those in power are not willing to do it.
(I think most of humanity would be okay with, say, heavily taxing super wealth to protect nature, fight climate change, or feed the poor. They’d also be okay with regulating pollution or not allowing 3M to ruin Belgium’s groundwater.)
by port11
5/31/2026 at 3:40:21 PM
I agree with all these observations.This is the best argument for successionism IMO. If you can be confident that you are creating a BDFL that is genuinely better than human leaders (a quite low bar) then it seems a good trade, unless you are quite optimistic about humanity’s prospects for improvement.
The problem of course is how to be confident you are creating a good BDFL and not handing control of humanity’s future to an indifferent-at-best, deceptive/malicious at worst successor.
An especially thorny problem - even supposing success on all these difficult alignment problems; supposing Claude Omega really is super-rational / super-moral, and we all vote to make them president of Earth. Things might go great for a while. How would you be confident that a self-modifying agent can retain its values as it grows and re-trains itself?
This is where the LessWrong folks’ explorations into decision theory really come to bear: morality in the face of self-modifying agents becomes very weird. A lot of human moral intuitions break when the principals are able to modify their own code. (See Timeless Decision Theory for an attempt to solve these problems.)
I think the summary is, if you hand control over to a self-modifying AI anything like our current systems, it will go very badly.
by theptip
5/31/2026 at 3:45:44 PM
Any supposed "AI BDFL" will be controlled by a human. The base concept is inherently flawed.by justinlivi
5/31/2026 at 4:00:20 PM
No.. that’s not what AI succession means. This all supposes a powerful and capable enough AI entity that there is no human in control.Rather than simply asserting that your interlocutors are wrong, you are welcome to advance an argument for why you think this is the case.
by theptip
5/31/2026 at 5:16:09 PM
If you keep things in the realm of thought experiments, you're allowed to say "OK, play along with my hypothetical here."Once you're positing things that will/might happen in reality, you need to be willing to accept criticisms of the feasibility of your premises. More, you need to be able to provide evidence of the feasibility of your premises.
You certainly don't get to just tell people, "No, AI succession means AIs will take over. That means you aren't allowed to claim AI's can't take over!"
As things stand, there is zero evidence that any current system is on a trajectory to produce an AI that can operate meaningfully without humans in charge. Even the most sophisticated agentic systems still have humans providing the motivation, the instructions, and all the various linkages that make them capable of acting (even if the motivation and instructions given are purposefully vague or open-ended).
by danaris
5/31/2026 at 4:07:40 PM
The burden of proof is not on us to show why wishful, magical thinking is wrong.People who claim that AI will take over can’t point to any evidence. All they have is speculation.
by satisfice
5/31/2026 at 5:55:39 PM
I would rather have an AI overlord with no morality at all than AIs "align"ed to the values of tech billionaires like Altman or Amodei.by pushkin_05
5/31/2026 at 3:13:00 PM
> It's possible that human morality is not sufficient to solve the problems with the world. It wasn't designed to operate at that scale.> i.e. it would look immoral to us.
I doubt it actually solves the problem then. Especially because I would be willingly to bet it would be almost impossible to get large enough groups of people to agree what the actual problem is.
by Avicebron
5/31/2026 at 7:22:25 PM
Sounds like it depends on how you would prefer to have your standards of morality be set, and/or enforced.Whether the arbiter is a human or a machine, the appropriate training set has to come from somewhere :\
Until then there's got to be some defect remaining to be overcome before a truly adequate training set can be accessed that can help with morality.
Which appears to be elusive enough a problem to potentially outlast a species, as we have seen so far.
As to the training sets, people are just using what they already had, some since prehistoric times, because so many of them don't have anything better.
by fuzzfactor
5/31/2026 at 5:59:25 PM
> It's also quite likely that a "superior" morality — the one that's actually built for a global scale — would be incomprehensible (likely requiring superintelligence to even access), and alien or disturbing from our vantage point.Congratulations, you just reinvented the ineffability of God.
by BobaFloutist
5/31/2026 at 3:32:19 PM
*not yet willing to do it... But look at the generations coming up who are dealing with this technological capture of humanity. They don't like it.by daveguy