math's way of asking what you actually want, then making you put a number on it.
means a mathematical function that assigns a numerical value to outcomes, representing how much an agent prefers one over another.
from born in 18th century economics with bernoulli's attempt to explain why people won't bet everything on a coin flip, formalized by von neumann and morgenstern in 1944 to prove rational choice under uncertainty could be modeled with numbers.
von neumann morgenstern — 1944 axioms proving rational preferences can be numbered.
bentham's felicific calculus — 1789 attempt to literally quantify pleasure and pain.
reinforcement learning reward — chatgpt style models trained to maximize a scalar function.
expected utility paradoxes — allais paradox, 1953, showed humans routinely violate the model.