Human neuroimaging data is fascinating, but most of it is done from a distance (EEG) or by proxy (fMRI), making it necessary to infer neural processes from rather crude data. In some cases, we take an EEG or fMRI pattern and give it a name which implicitly or explicitly alludes to a mechanism, without enough evidence that there is a one-to-one relationship. Our lab pointed out some of these pitfalls (in which we ourselves have sometimes fallen) and how to correct them. We found that a scalp increase of high frequency (gamma) EEG which was taken to reflect gamma oscillations was in fact an artifact of ocular muscles activity at the onset of saccades; that inter-trial coherence in response to rhythmic sequences which was taken as reflecting oscillatory entrainment is just as prominent in situations that are not conducive to oscillatory entrainment; and that "cross-frequency coupling" which was taken as a signature of two oscillatory processes could be a signal processing artifact resulting from non-sinusoidal waveforms.
Prediction, change detection, and attention
A critical function for complex behavior is the ability to focus attention, investing perceptual, cognitive function, and motor resources into the activity with the highest current priority. However, doing this we risk missing either critical threats or enticing opportunities outside the focus of attention. Thus, flexible behavior requires that we keep a surveillance mechanism sensitive to background deviations from previously established regularities (which can be conceived also as prediction violations). Such a mechanism is indexed presumably by the mismatch negativity (MMN) event related brain potential (ERP). The MMN is elicited by changes in the environment even when the stimuli are outside the attentional focus and is thus considered pre-attentive. It is most easily found for acoustic (auditory) deviations. Investigations using EEG, MEG, intracranial ECOG, and fMRI, suggest that the main sources of auditory MMN is in the superior temporal gyrus, but we and others provided also evidence for a frontal generators (Deouell, 2007; Deouell, Bentin, and Giard, 1998; Shalgi and Deouell, 2007).
Shalgi and Deouell (2007) confirmed that MMN is robustly elicited by deviants occurring on one side of space (in free field) even when attention is strongly focused on a difficult task on the other side. However, if the deviants in the unattended task are on the same dimension (e.g. pitch or location of sounds) as the targets in the attended side, the temporal component of the MMN (as defined by current source density) is strongly attenuated. In contrast the frontal scalp component is unaffected. This result confirms that the mismatch response is on one hand quite robust to general within-modality attentional effects (even if not completely encapsulated, Haroush, Hochstein and Deouell, 2010; Haroush, Hochstein and Deouell, 2011), but on the other hand it is susceptible to competition for processing resources (as suggested by Sussman et al., 2003). It places this competition at the temporal component of the MMN, providing the first evidence for a functional separation between the frontal and temporal components of the MMN that we are aware of. The results did not shed light though about the specific role of the frontal cortex in mismatch detection, as neither attention load nor processing competition affected this component.
An interesting answer came recently from recordings done intracranially, in patients suffering from intractable epilepsy who are candidates for surgical resection of the epileptic focus. These patients are implanted with grids and strips of electrodes on the surface of the brain, which allow better localization of the epileptic spikes. The patients graciously volunteer their time to participate in experiments, providing invaluable electrocorticographic (ECOG) data. Several years ago, we (Edwards, Soltani, Deouell, Berger and Knight, 2005 ) found increases in broadband high frequency (BHF) responses to deviations in a simple MMN paradigm. Since then, numerous studies established the BHF response in multiple tasks and brain regions, as a signature of cortical activation, related to population firing rate. We now measured this signal in 5 patients, while exposed to task-irrelevant sequences of 80% standards and 20% deviants. One condition was completely predictable as every 5th sound was a deviant. The other condition was hard to predict as deviants occurred irregularly (with the constraint of at least 3 standards before each deviant). Both the temporal cortex and the ventrolateral prefrontal electrodes revealed BHF increase to deviants, but a clear dissociation emerged between the regions. While the temporal cortex responded similarly to predictable and unpredictable deviations, the frontal cortex responded with BHF increase only to unpredictable ones! That is, the temporal cortex was myopic, and only considered the most recent history, while the frontal cortex considered the 'big picture', i.e., the statistical structure of the sequence as a whole.
We recently found evidence in the frontal cortex for actual proactive prediction (or anticipation) of the deviant, just prior to the occurrence of the deviant, in the predictable condition (Dürschmid et al. 2018). This pre-deviant anticipatory signal (in the form of reduction of BHF amplitude before predicted deviants) correlated with the post-stimulus response to the deviant – the more the pre-deviant BHF was reduced, the less likely was a BHF response to the expected deviant. This was a rare evidence that predictions (in the common sense of anticipating a future event) actually takes place. Viewed from the perspective of the predictive coding framework, the findings suggest that two levels of the mismatch processing hierarchy maintain two parallel predictions simultaneously, and these predictions may in some cases be contradictory. For example, following 4 standards in the regular condition, the temporal cortex seems to predict more of the same (another standard) and elicits a mismatch response ("prediction error signal") when this prediction is violated. In contrast, for the very same stimulus, the frontal cortex actively expects a deviant, and the mismatch response is therefore absent (no prediction error). It is important to understand how such parallel predictions are maintained in the system, and why predictions do not trickle down from higher level nodes to lower level nodes.
Perception, Attention and Conscious awareness
Which processes are required for conscious awareness to emerge, and which cognitive processes require awareness? Our studies on these questions range from studies of patients with unilateral neglect who lack awareness of their contralesional side because of their brain damage, through studies of healthy individuals who are unaware of events or stimuli either because of lack of attention or because of deliberate masking. We have used EEG and ECOG to gain insight into these questions.
The lesson we have learned from patients is that being able to represent space is critical for being aware of events occurring in that space (see Deouell (2002), for summary of this idea). We (Deouell, Deutsch, Scabini and Knight, 2008) later showed that patients with unilateral neglect following right hemisphere stroke are susceptible to the Scale Illusion, in which sounds from both ears are merged into a ‘tune’ although they are only aware of stimuli arriving to their right ear. Thus, ‘neglected’ information can become part of conscious awareness if it can misattributed to another location in space. In a way, it suggests that the allesthesia seen in neglect is not so much a deficit as a way to ‘salvage’ information from oblivion.
One of the most important functions ascribed to the conscious state is integration of information, across space, time, and modalities. In a series of studies we \ looked into integration of an object with a complex, natural scene. In Mudrik, Lamy & Deouell (2010) and Mudrik, Shalgi, Lamy and Deouell (2014) we used ERPS to look at the temporal evolution of scene perception when scenes include a semantically incongruent objects, when no a-priori expectations are involved. The results confirm that contextual congruity affects scene processing starting from ∼300ms or earlier, and that this early electrophysiological congruity effect indeed reflects context violation processing, rather than indexing a mismatch between expected vs. actual events, or between prepared vs. correct responses. They also suggest that contextual information may affect object model selection processes, and influence later stages of semantic knowledge activation and/or decision-making. Next, we (Mudrik, Deouell & Lamy (2011)) presented a scene separately but simultaneously to the two eyes, except that a critical object in the scene was congruous in one eye and incongruent in the other, creating binocular rivalry for that object (while the scene was static). The objects alternated in dominating conscious perception, but the incongruent object lingered in consciousness for longer. Yet, we found no evidence that the incongruent object captured attention preferentially. In another line of studies, we (Amihai, Deouell & Bentin, 2011) investigated adaptation effects when faces were masked by continuous flash suppression (CFS). Our results suggest that adaptation to gender and race depend on the duration of conscious awareness of the face, rather than on the full duration of time the subject was exposed to the face. Previous studies showed that categorical information can be extracted without awareness, but it seems this may not extend to the subordinate level. Finally, in Mudrik, Breska, Lamy & Deouell (2011)), we masked whole scenes using continuous flash suppression (CFS) and measured how long it takes the scene to break the suppression and emerge into awareness. We surmised that a difference in this time between congruent and incongruent scenes will suggest that the congruity (and thus the integration of object and scene) occur without awareness. Indeed incongruent scenes emerged earlier, suggesting integration without awareness for complex scenes. However, further studies failed to replicate the effect, and so the jury is still out with this paradigm. Still, reviewing the literature, it is clear that some level of integration does take place without awareness (Sklar, Deouell & Hassin, 2018).
Conscious awareness and action monitoring
Conscious awareness is not reserved only for perceptual tasks. What is the role of awareness in action monitoring? We investigate awareness of performance errors using ERPs. Shalgi, Barkan & Deouell (2009), showed that the response-evoked Pe was clearly associated with awareness of the errors, while the earlier ERN\Ne seemed independent of awareness. However, single trial analysis suggested that the Pe might be a delayed P3 response – i.e. related to stimulus appraisal rather than to error-awareness per se. We (Shalgi and Deouell, 2012) then revisited the premise that the ERN\Ne is elicited without awareness. We wondered if the ERN for putatively unaware errors was actually due to trials in which the subjects were aware of the error but with lower confidence, and therefore did not report the error. By asking subjects to bet money on their error judgment, we could choose as unaware errors only the error trials in which the subject were so sure that they did not err, that they were willing to risk a lot of money. Taking these trials as true unaware errors we found no evidence for ERN\Ne without awareness! However, in reviewing the literature, we (Shalgi and Deouell, 2013) suggested that "…the initial error signal indexed by the ERN/Ne does not reflect error awareness per se, but rather it is a prerequisite of this process, and thus, correlated with it. Possibly, if some Ne is generated, a second, compound internal error signal based on additional sources of information (including later events, such as proprioceptive feedback, autonomic responses and sensory input) may exceed the threshold for error awareness."
To further look at the role of conscious awareness in action control, with Ariel Furstenberg and Haim Sompolinski, we addressed a mundane but curious situation called 'picking' (Furstenberg et al., 2015). This happens when we are faced with a simple choice between two options that are practically identical for us – two identical cans of coke for example. Why are we not stuck like Buridan's ass who perished between two identical piles of hay as it couldn't make up its mind which to eat first? Leibniz suggested no 'free will' needs to be invoked to get out of this deadlock because this apparent symmetry outside is usually broken by some asymmetry of internal parameters, of which we are unaware, and this asymmetry tips the scale this way or the other. Our subjects were asked to choose between pressing a button with their left or right hand. In some trials they had to choose according to a right or left instructive arrow, whereas in other trials they were cued with a + sign to choose as they wish ("free choice"). Unknown to the subjects, just before the cues (the arrows or +), we flashed a masked arrow. Even though the subjects were not aware of this, the masked arrow had its influence. Subjects were slowed to respond to an arrow if it was preceded by a subliminal arrow pointing in the other direction. We used the lateralized readiness potential (LRP) to understand the mechanism. The LRP indicates the intention to move the right or left hand. When the masked arrow pointed in one direction and the arrow cue was in the other direction, the LRP nicely reflected a change of intention – first an intention to move the hand directed by the masked, subliminal arrow, followed by a change of intention and a button press by the hand instructed by the final cue. Thus we defined a neural signature of 'Change of Intention' (ChOI). More importantly, the signature of ChOI was also seen in those free choice trials in which the subjects pressed the button which did not correspond to the direction of the masked cue! That is, we obtained evidence that the subliminal cue instigated an intention to move one hand, thereby creating the internal asymmetry Leibniz referred to, and nevertheless, this was 'freely' over-ruled and the subject pressed with the other hand. We have recently replicated the result in groups of young and older adults and continue to explore this phenomenon using computational modeling ()
Perception beyond onsets
Much has been learnt in the last few decades about the organization of visual processing in the brain. However, peculiarly, most of the evidence comes from responses to onsets of stimuli – that is, moments of change. Experiments typically involve abrupt appearance of visual stimuli on the screen, which quickly disappear. While this elicits robust response during the first ~200 ms post-stimulus), aware perception continues beyond this immediate response. Consider for example staring at a still statue for a few seconds. What is the brain activity which correlates with this persistence of the object in our perception? In a recent set of experiments we examined this question. First, using electrocorticography in patients implanted with grids of electrodes (see section on Perception, Attention and Awareness), we tested the responses to images of different categories, presented for 5 different durations (from 300 ms to 1500 ms). The results revealed a new organizational principle of the visual cortex – broadband high frequency (BHF) in early visual cortex (V1-V3) persisted for as long as the stimulus was presented, but this 'duration tracking' became less and less reliable at the single trial level moving forward along the ventral visual stream in the inferior temporal cortex. In the anterior part of the stream, the category of the stimulus was well represented in the onset response, but this activity did not reliably persist for the duration of the stimulus. Thus, in contrast to some theories, it seems that the sustained perception of a specific object is not a result of activity in high level visual areas selective for the object category. Rather, it relies on a concerted activity of early non-specific-but-sustained neural activity, and late specific responses which do not last. In a few healthy subjects we found comparable duration tracking HFB activity using EEG, highly focal to one or two occipital electrodes. We are currently investigating this response, which gives hope that we can address the question using non-invasive measures.
Intriguing dissociations between activity in early and high order visual cortex was also found when we looked (with Tal Golan and Rafi Malach) at the response during blinks and saccades. Blinks and saccades provide unique dissociation between the visual input and aware perception: despite the fact that the input is abruptly reduced (during blinks), or distorted (during saccades), we are unaware of any interruption. In contrast we are aware when we voluntarily blink, or when the scene moves. As we found in the studies of sustained perception, early visual cortex reflected the input with high fidelity –activity dropped during voluntary and spontaneous blinks, as well as darkening of the room, and then resurged when the input resumed. The findings in higher order visual cortex suggested that spontaneous blinks' limited visibility compared with darkenings is correlated with suppression of blink-related visual activity transients, rather than with "filling-in" of the occluded content during blinks.
Decision making and biases
What happens when one has to blindly choose between two options, with no information about which is a better one? We know that when the outcomes are revealed, subjects' satisfaction is not determined by their outcome alone, but also by comparing it to the unchosen outcome (I am happy to gain 10 Euros but even more so if I could have lost 10 Euros had I made a different choice). Using the feedback related negativity as a putative index of 'satisfaction' Dvorah Marciano found the surprising Alternative Omen Effect (ALOE). It turned out that if the two outcomes are not revealed together, but, rather, the unchosen (alternative) outcome is revealed first, subjects are more satisfied if the two outcomes are gains, then if their outcome is a gain, and the alternative is a loss. Very different than the typical outcome comparison literature. In a long series of experiments we discovered why – even though the two outcomes are uncorrelated and the subjects perfectly know it, the subjects see a positive alternative outcome as an omen, predicting that their own outcome will be bad. That is, they experience an illusory negative correlation between the outcomes. We showed that this happens even in situations that cannot possibly be conceived as zero-sum games. Our results lead us to believe that the reason is a belief in limited good luck: if the alternative outcome is good, then the other outcome is less likely to be good as well. This may have fundamental effect on people's behavior and the way they negotiate.
Preattentive processing of auditory space
Studies of patients with unilateral neglect suggested that one putative prerequisite for awareness of external events is the encoding of its spatial location (Deouell, 2002). In a series of studies we have begun to investigate the processing of the location of unattended sounds. Do we actually encode the location of background sounds, and if so, at what resolution? We used the MMN event-related potential, as well as fMRI, to investigate these questions. In Deouell et al. (2006) we showed, using the MMN, that indeed location is encoded pre-attentively, with a resolution of at least 10 degrees, and that the magnitude of the change in location is registered in the amplitude of the MMN. We later extended this to both sides of space, up to a laterality of 60 degrees, and relating to changes medially or laterally. A series of fMRI experiments (Deouell et al. 2007), showed that the human planum temporale responds to spatial variation even when subjects are not attending the auditory stream. Using the fMRI adaptation rational we recently showed that the Planum Temporale neurons are tuned simultaneously to spatial locations and pitch (Shrem & Deouell, 2014). We showed that the representation of space is both in head-centered and environmentally- (or body-) centered space (Shcechtman et al., 2012), and that location-specific repetition suppression is modulated by audio-visual interaction (ventriloquist effect; Shrem et al., 2017).