Striatal interneurons and reward prediction errors (Commentary on Apicella et al. )

Cortical-basal ganglia circuits play a vital role in the organization of thoughts and behavior. However, the specific computational processes achieved by these circuits remain unclear. Part of the problem is that the main input nucleus of the basal ganglia, the striatum, receives inputs from very wide regions of cortex and thalamus; electrophysiological studies of single striatal neurons show correspondingly diverse patterns of activity, that often defy ready classification and analysis. However, several specific subtypes of basal ganglia neurons seem to have simpler, more stereotyped patterns of firing. The best known are the midbrain cells that provide dopamine to the striatum (and elsewhere); in an enormously influential series of papers Wolfram Schultz and colleagues (e.g. Schultz, 1998) argued that these provide a unified reward prediction error signal that drives reinforcement-based learning. But another cell class that has also received much attention are the ‘tonically active neurons’ (TANs) encountered in monkey striatum. These are presumed to be cholinergic interneurons, based on the electrophysiological properties of such interneurons in rat in vitro studies. Normally firing at a moderate and steady pace, they show characteristic brief firing pauses in response to a range of salient events. Like dopamine cells, TANs are few in number but nonetheless appear to provide important control over striatal synaptic plasticity, widely considered to be a major substrate of reinforcement learning. While several groups have observed the characteristic TAN pause response, debate continues over several issues that are key to understanding the computational role of this control signal. The TAN pause response is known to be dependent on intact striatal dopamine (Aosaki et al., 1994) – so is it just passing along a signal or does it provide a distinct message to dopamine? Is this message reward-specific, or does it also occur with unexpected aversive events? And does it encode both ‘positive’ errors (an unexpected salient event) and ‘negative’ errors (omission of an expected event), to allow bidirectional control over plasticity? The paper by Apicella et al. (2009) in this issue of EJN, contributes to this ongoing debate. The authors employ a behavioral task in which cues predict forthcoming rewards with varying probability, so that the contribution of expectations to neural firing can be assessed. While this approach has been used before, Apicella et al. (2009) varied the probability of reward across blocks of trials within the same session, rather than using cue-reward probabilities that are fixed over thousands of trials. In line with one aspect of standard reinforcement learning theory, they found that the TAN pause response to reward was diminished when the reward was fully predictable. They then looked at omission of rewards, and found that the TANs split into two groups: one group that increased firing shortly after the expected reward time and another that decreased firing. In addition, even those TANs that did increase firing to reward omission did so with a variable timecourse, in contrast to the stereotypical pause response. This suggests that TANs are not serving as a unified, bidirectional signal encoding both positive and negative reward errors. These results extend a growing body of work indicating that both the cholinergic and the dopaminergic basal ganglia control signals are not as simple and unified as once thought. For example, many TANs care about the spatial location of instruction cues (Ravel et al., 2006), and TANs in the caudate part of striatum seem to care more about the onset of motivationally relevant cues than TANs in putamen, which care more about cues instructing movement onset (Yamada et al., 2004). On the dopamine side, recent papers have shown that many presumed dopamine cells fire more to aversive cues than appetitive cues (Joshua et al., 2008; Matsumoto & Hikosaka, 2009). The challenge for the field is to determine whether such variation reflects a multiplicity of functions for these neurochemical signals, or if there remains a single, underlying fundamental computation that these signals help to accomplish.