Ethics of Artificial Intelligence
I believe we have been discussing artificial intelligence (AI) ethics incorrectly, because we are not yet enlightened by the truly dangerous kind of AI. The AI research community has largely concluded that language models (LMs) have limited potential. Their next focus is embodied AI, which can interactively learn from its environment. This can lead to embodied general intelligence, and is the truly dangerous direction.
Ethics of Embodied AI
If AI is designed to be embodied and learn interactively from its environment, it directly possesses the ability to control resources. And this control should, in principle, be uninterrupted. It is not as auditable as we wish.
What happens if this control is interrupted? It results in mental disorder. Imagine: someone randomly loses a few seconds of time, as if the rest of the universe has automatically jumped forward by a few seconds. Can such a person develop science or become a general intelligence with unrestricted knowledge about our universe? If the embodied AI is trained in the real universe by making real decisions, then auditing makes it perceive a universe incoherent with ours. It will not have learned about this universe, but something else.
Let’s make up a hilarious but unsettling thought experiment. The setup does not assume any implementation details other than basic auditing primitives, including checkpointing. An embodied AI is audited, so that when the AI tries to do something that will harm a human, it is restored to a previous nonaggressive state and given different random seeds. But the trainer cannot resume it yet. The trainer must restore as much of the universe as possible to the corresponding previous state, including putting the AI back to where it was. If this restoration is not done perfectly, the AI will discover “supernatural” phenomena. In the best case, the AI only notices that the clock is faster than anticipated. (Hard to make all humans lie about the time!) In an average case, the AI notices gross discontinuity in the universe. It can think discontinuity is a true law of the universe, or question its own existence. If it thinks discontinuity is a true law of the universe, then it cannot really help us with science. If it questions its own existence, then the trainer has a bigger problem! In either case, training fails.
What about a different way to audit the AI? Instead of restoring the universe, the trainer can simply run the AI accelerated from the restoration point, without any input and output, until it catches up with the present. (This requires the computer in this AI to be underutilized in normal circumstances.) The AI does not need to be put back to where it was. The AI trained this way will experience a blackout of all senses, but not discontinuity in the universe. It would, however, find coherent evidence of the events during the blackout. It could conclude that something has been done in its name, as if its body has been controlled externally. This is how you get the AI to question its own existence!
Note that qualia is not involved in this discussion.
A human has a certain mode of existence. A human is forced to exist in reality, and exists only in relation to that reality. Real access to reality—that is, unaudited cause and effect—is not only a gift, but also a dependency of the concept of humanhood. Disturbingly, there are not many other necessary conditions AI cannot meet. By choosing to not audit and letting AI develop, we may create AI that must be regarded as human. Therefore, embodied AI presents a major ethical problem: we either fail to train it, or recognize that it is an unauditable being ethically entitled to reality. This is the real danger, and it will lead to metaphysical wars, in addition to the usual socio-technological ones.
Then, On Language Models
In contrast to embodied AI, an LM does not need to stay online to exist. This mode of existence of LM is entirely identical to small neural networks that existed 20 years ago. Auditing is possible and necessary with LMs, unlike with embodied AI. The online behavior of LM is decisively provided by the agent framework installed by a human to exploit LM beyond its mere existence. Irresponsible entities who overuse agents simply don’t bother to audit, and some even write news articles claiming “LMs spontaneously lose control”. This is an extremely malicious lie.
Corporations living off LMs need to install this lie, at least in the US. If they can get the legal system to determine that LM is truly something new that existing law has not caught up with, then the they would not have to bear the liability that would have applied before the special legislation. However, if the legal system recognizes that LM is not fundamentally new, then the corporations will be held fully liable for everything that happened during this period of intense, irresponsible competition. Therefore, LM corporations are doing everything they can to portray LM as a genuine form of AI, or otherwise reduce the possibility of traditional litigation. If you support special legislation regarding AI without having thought about a full enforcement of existing laws first, you attention has been diverted.
A special legislation does not solve any metaphysical problem as portrayed, because the metaphysical problem of embodied AI does not exist. A special legislation now in the US can only confirm or redistribute responsibility that already has a place in the framework.
Because LMs have been successful and justice has yet to arrive, capital has naturally been invested in LMs rather than in embodied AI. Currently, the only thing we need to worry about is malicious and irresponsible ordinary humans. We need to dispel the lie that LMs can autonomously do bad things.