In reinforcement learning (RL) applications, random perturbations influence the exact amount of reward received. A typical algorithm predicts the average reward across multiple trials, and uses this prediction to decide how to act. In our latest work, we show it is equally possible to model not only the average but also the full variation of this reward, what we call the value distribution.  This not only results in RL systems that are more accurate and faster to train than previous models, but more importantly opens up the possibility of rethinking the whole of reinforcement learning.