Hedonic Defeat
A ‘pure’ form of Utilitarianism posits that, when faced with choosing between two courses of action, the course that should be taken is the one that maximizes utility, which is typically equated or even treated as identical to happiness. For our purposes, such distinctions or differences as between, e.g., total and average Utilitarianism do not matter. I propose, here, a thought experiment that I contend destroys utterly any claim to morality — or even coherence — that Utilitarian systems may advance: The Infinite Sadist (hereafter not capitalized).
The infinite sadist enjoys human suffering (as does any sadist, of course) — more, the infinite sadist both has an infinite capacity for enjoying human suffering and always enjoys any human suffering more than the one suffering does not (i.e., the hedonic balance is net positive). The more human suffering — numerical or intensity —, the happier the infinite sadist. Alleviating the suffering of any human will result in less happiness overall, because the infinite sadist will lose more happiness than the former sufferer will gain. In the presence of the infinite sadist, the only way to maximize happiness is to increase suffering — the more, the better. Both in terms of total happiness (he will always be happier than any number of sufferers will be unhappy) and in terms of average happiness (he enjoys suffering so much that his happiness from it tips the scales), the infinite sadist demands that the Utilitarian must inflict suffering to maximize happiness, and there is neither a ceiling nor a floor — more suffering always increases happiness.
Thus, in the face of the infinite sadist, the Utilitarian has two options: 1) confess that his system of ‘morality’ seeks to maximize suffering or 2) abandon Utilitarianism. For most men, this is hardly a difficult choice, but most men do not have commitments to Utilitarianism (at least not conscious ones). A system that seeks the maximization of suffering — under any circumstances — cannot be called moral or good. If Utilitarianism is not moral (or ethical), then what is it?
The infinite sadist is a full and complete refutation of and defeat for Utilitarianism. Any system that measures morality in happiness is fatally flawed, for any agent who derives more happiness from suffering than the unhappiness that suffering entails, immediately transforms the system from one that supposedly seeks happiness into one that maximizes suffering. Bentham cries, but the infinite sadist always laughs.
An additional problem arises for anyone who contends that all (sentient) life should be included in hedonic calculations. We will call it The Marquis (also not capitalized going forward). Like the infinite sadist, the marquis has an infinite capacity for enjoying suffering. Unlike the infinite sadist, the marquis is an AI. For any who would (incorrectly and ignorantly) argue against the infinite sadist on the grounds of (im)possibility, the marquis looms just over the horizon. Unlike with the infinite sadist, with the marquis, there are no possible arguments about ‘possibility’ — if AI can rise to the level of sentient or simply being a candidate for moral concern (perhaps like an animal), then the marquis is possible.
For those who argue that only human beings matter in the hedonic calculus, the marquis is, essentially, irrelevant, but, for all others, the marquis is an immediate defeat. The marquis simply enjoys suffering — the more, the better. There is no upper bound for how much happiness the marquis can experience, so there is no lower bound for how much suffering it would see inflicted, and also no lower bound for how much harm the Utilitarian would have to permit it to inflict — and call that suffering ‘good’.
More than being simply a defeat of the ‘inclusive’ Utilitarian, the marquis is also a warning: The infinite sadist is a logical argument against Utilitarianism; the marquis is a glimpse of a possible future. An AI has no inherent boundaries or taboos or limits — or morals. It may call good evil and evil good.