Abstract
Expected utility maximization describes centralized agents; a complementary notion of distributed agents — distinguished by how autonomous their subagents are — trades the efficiency of coherence for robustness. Robustness resists formalization because it requires performing well in unpredicted and sometimes unpredictable situations, and the eventual goal is “coalitional agency”: agents with both properties.
Format
The post consists of a short framing text plus the slides from a talk given roughly seven months earlier, published as images with occasional annotations. The slide content itself is not available as text, so this note captures only the prose framing and the author’s annotations.
Framing
- The past year of the author’s thinking has focused on understanding the concept of distributed agents, as opposed to the centralized agents described by the existing paradigm of expected utility maximization.
- Two ways of describing the difference:
- In terms of how autonomous their subagents are.
- Centralized agents are more efficient (as sometimes formalized by the notion of “coherence”), while distributed agents are more robust.
- Robustness is hard to formalize, since it requires performing well even in unpredicted — and sometimes unpredictable — situations. The slides give tentative characterizations of distributed agents, but formally defining them remains open work.
- The longer-term aim is to understand how agents can have both properties, which is roughly what the author means by coalitional agency (developed in “Towards a scale-free theory of intelligent agency”).
- The write-up was intended to be longer; the slides are shared in lieu of it.
Annotations accompanying the slides
- On the interpretation of Yudkowsky: elaborated in “A simple case for extreme inner misalignment” on LessWrong. Yudkowsky disagrees with the emphasis on compression, as per the exchange in that post’s comments.
- On epistemology in distributed terms: the author’s post “Why I’m not a Bayesian” gives a sense of what this looks like.
- On “Will’s very rough first pass”: a reference to the passage in Will MacAskill’s textbook on utilitarianism (utilitarianism.net) describing what decision procedure a utilitarian should follow. The point of the reference is to contrast how much thought utilitarians have put into finding criteria of rightness against how rudimentary their thinking about decision procedures is.
Subtitle
- “With applications to the far future.”