Acting Under Moral Uncertainty

Moral Uncertainty

Moral uncertainty arises when you are uncertain which moral theory is correct. Even if you're confident in your own moral reasoning, you might recognize that other thoughtful people disagree, suggesting some chance you're wrong. How should you act when unsure about the fundamental principles of ethics?

Aggregation Methods

• Maximizing Expected Choiceworthiness (MEC): Compute the expected value of each action weighted by your credence in each theory: CW(a) = Σ P(T) × U_T(a). Choose the action with highest choiceworthiness. This is analogous to expected utility theory under empirical uncertainty.
• My Favorite Theory (MFT): Simply act according to whichever moral theory you find most credible. This ignores the possibility you're wrong and can lead to moral fanaticism.
• Variance Voting: Weight theories by credence but penalize high-variance actions that are extremely good under some theories but terrible under others. Encourages moral hedging.
• Parliamentary Model: Imagine theories as legislators voting on actions. Each theory gets votes proportional to your credence. Can model bargaining and vote-trading dynamics.

The Normalization Problem

A key challenge: how do we compare utilities across theories? Utilitarianism might assign 100 utils to saving one life, while deontology assigns 50 to keeping a promise. Are these numbers commensurable?

• Cardinal approach: Take utility numbers at face value. Assumes intertheoretic comparisons are meaningful (controversial).
• Ordinal approach: Normalize each theory's utilities to the same scale (e.g., 0-100). Respects only rankings within each theory. More defensible but loses information about intensity.
• Variance normalization: Normalize by each theory's standard deviation. Makes comparisons based on how much theories "care" about differences.

Moral Hedging

When facing moral uncertainty, you might choose actions that perform reasonably well under multiple theories rather than optimizing for your favorite theory. This is moral hedging - similar to financial hedging, you sacrifice some expected value to reduce risk of moral catastrophe.

Example: You're 60% sure utilitarianism is right, 40% sure deontology is right. Utilitarianism says to break a promise to save one life. Deontology says keep the promise. Pure expected choiceworthiness might favor breaking the promise, but you might hedge by keeping it to avoid doing something deeply wrong if deontology is true.

The Fanaticism Problem

Some moral theories assign infinite or near-infinite value to certain outcomes (e.g., preventing eternal suffering). Even tiny credence in such theories can dominate decision-making. This leads to fanaticism: any non-zero probability of infinite value trumps all finite considerations.

Possible responses: bound utilities, use parliamentary models where extreme theories can be outvoted, or adopt variance penalties that discourage actions with extreme outcomes.

Philosophical Issues

• Metaethical assumptions: MEC assumes moral realism (there's a fact about which theory is correct). Constructivists or expressivists might reject this framework entirely.
• Peer disagreement: Should widespread disagreement among ethicists increase your moral uncertainty? Or should you trust your own reasoning?
• Normative vs descriptive: Is moral uncertainty special, or is it just regular empirical uncertainty about moral facts?
• Decision procedure vs criterion of rightness: Even if MEC is the right decision procedure under uncertainty, the correct criterion might still be one specific theory.

Applications

Moral uncertainty is especially important for high-stakes decisions: effective altruism prioritization (global health vs animal welfare vs AI safety), policy-making with ethical tradeoffs, personal career choices with moral implications, and existential risk mitigation where different theories radically disagree.

Further Reading

• MacAskill, W., Bykvist, K., & Ord, T. (2020). Moral Uncertainty. Oxford University Press.
• Lockhart, T. (2000). Moral Uncertainty and Its Consequences. Oxford University Press.
• Sepielli, A. (2009). "What to Do When You Don't Know What to Do." In Shafer-Landau (Ed.), Oxford Studies in Metaethics.