Diversity in Harmony: Proceedings of the 31st International Congress of Psychology, Diversity in Harmony: Proceedings of the 31st International Congress of Psychology

About the Editors

Kazuo Shigemasu is Professor Emeritus of the University of Tokyo, Japan, and visiting Professor of Psychology at Keio University (Tokyo), and has held faculty appointments in psychology at the University of Tokyo, Teikyo University, Tokyo Institute of Technology, and Tohoku University. His research focus is methodology in psychology, particularly based on the Bayesian statistical approach. Shigemasu has served as president of the Japanese Psychological Association (JPA), the Behaviormetric Society (MS), and Japanese Association for Research on Testing (JART).

Sonoko Kuwano is Professor Emeritus of Osaka University, Japan. Her main research focuses on environmental psychology. She is a member of the Science Council of Japan. She has served as a member of the Executive Committee of the International Union of Psychological Science, Vice President of the International Commission for Acoustics, President of the Acoustical Society of Japan, and President of the Japanese Society for Music Perception and Cognition. She received Commendations for Contributions in Environmental Conservation from the Minister of the Environment in 2006.

Takao Sato is Dean and Professor of Comprehensive Psychology at Ritsumeikan University, Osaka, Japan. Formerly Professor of Psychology at the University of Tokyo, his research is mainly concerned with visual and auditory perception, especially visual perception of spatio‐temporal patterns, and of motion and depth. He was President of the Japanese Psychological Association, President of the Vision Society of Japan, and President of the Japanese Psychonomic Society.

Tetsuro Matsuzawa is Distinguished Professor at the Kyoto University Institute for Advanced Study (KUIAS), Kyoto, Japan. His research focuses on the cognition and behavior of chimpanzees, both in the wild and in the laboratory. Matsuzawa is the former President of International Primatological Society, and the Editor‐in‐Chief of the journal Primates.

Notes on Contributors

Ralph Adolphs is the Bren Professor of Psychology, Neuroscience, and Biology at the California Institute of Technology (Caltech), USA. He directs the Caltech Brain Imaging Center, and his laboratory (emotion.caltech.edu) focuses on social neuroscience. Current research directions are to understand how emotions and social behavior arise in the brain, and predicting individual differences in these abilities from functional neuroimaging data. The laboratory includes studies of patients with focal brain lesions, fMRI, electrophysiology, and work in people with autism spectrum disorder.

Minoru Asada is Professor at the Department of Adaptive Machine Systems, Graduate School of Engineering, Osaka University, Japan. He is also a Division Chief of Systems Intelligence, Open and Transdisciplinary Research Initiatives at the same university. He has been a board member of the Japanese Society of Baby Science and the Japanese Society of Child Science since 2013. Since April 2017, he has been Vice President of the Robotics Society of Japan. He is also President of the NPO Leonardo da Vinci Museum Network, Osaka, Japan.

Ray Bull is Professor of Criminal Investigation at the University of Derby and Emeritus Professor of Forensic Psychology at the University of Leicester, UK. His major research interest is the investigative interviewing of suspects, witnesses, and victims, as well as witness memory, including voice recognition. He was elected Honorary Fellow of the British Psychological Society in 2010 and has been President of the European Association of Psychology and Law since 2014. He regularly acts as an expert witness and conducts workshops/training on investigative interviewing around the world.

Josep Call is a comparative psychologist specializing in primate cognition and cognitive evolution. He is Professor in the Evolutionary Origins of Mind (School of Psychology and Neuroscience) at the University of St. Andrews (UK) and Director of the Budongo Research Unit at Edinburgh Zoo. His research focus is on technical and social problem solving in animals with a special emphasis on the great apes, including causal and inferential reasoning, tool use, long‐term memory and planning, gestural communication, and mindreading.

Fanny M. Cheung is Vice President for Research and Choh‐Ming Li Professor of Psychology at the Chinese University of Hong Kong. Her research interests include cross‐cultural personality assessment and gender equality. After standardizing the MMPI and MMPI‐2 in Chinese societies, she noted the need for indigenous measures to fill the gaps in Western personality theories and assessment. She pioneered the combined emic–etic approach in personality assessment through the development of the Chinese Personality Assessment Inventory. This combined emic–etic approach is adopted in the development of other indigenous measures in South Africa and the Middle East.

Yann Coello is Professor of Cognitive Psychology and Neuropsychology at the University of Lille, France. He is the Director of the CNRS Laboratory Cognitive and Affective Sciences and President of the French National Committee of Scientific Psychology (CNFPS), a national member of IUPSyS. He has published numerous influential articles and books on the sensorimotor foundations of perception, cognition, and social interactions.

Frans B. M. de Waal is the Charles Howard Candler Professor of Primate Behavior in the Emory University Psychology Department in Atlanta, Georgia, USA, and director of the Living Links Center at the Yerkes National Primate Research Center. He is the author of numerous books including Chimpanzee Politics and Our Inner Ape. His research centers on primate social behavior, including conflict resolution, cooperation, inequity aversion, and food sharing. He is a member of the United States National Academy of Sciences and the Royal Netherlands Academy of Arts and Sciences.

Maria Eduarda Duarte is Professor of Psychology with the Faculty of Psychology at the University of Lisbon, Portugal. She is also director of the Masters course in Psychology of Human Resources, Work, and Organizations. Her professional interests include career psychology theory and research, with special emphasis on issues relevant to adults and the world of work.

Nancy Eisenberg Regents’ Professor of Psychology at Arizona State University, USA, is a developmental psychologist who studies social, emotional, and moral development, with primary interests in prosocial development and self‐regulation and their socialization. She is a past editor of Psychological Bulletin and Child Development Perspectives, and has received career contribution awards from the Association for Psychological Science, multiple divisions of the American Psychological Association, the International Society for the Study of Behavioral Development, and the Society for Research on Child Development. She has served as President of the Association for Psychological Science, Division 7 (Developmental Psychology) of the American Psychological Association, and Western Psychological Association.

Elaine F. Fernandez is the current Acting Head of the Department of Psychology, HELP University, Malaysia. She was a graduate of HELP University’s Bachelor of Psychology program, and obtained an MSc in Social Psychology (Distinction) from the University of Surrey, UK. She currently lectures in research and social psychology at HELP University, and is the convener for the Department of Psychology’s Centre for Diversity. At present, she is leading research projects tackling questions on Malaysian social identity, and the creation, maintenance, and consequences of social identification, both in general and in organizations.

Buxin Han is Professor of Psychology at the Chinese Academy of Sciences (CAS) Key Lab of Mental Health, Institute of Psychology, and the University of CAS, Beijing, China. He is Deputy Secretary‐General of the Chinese Psychological Society (CPS), Secretary‐General of the International Association of Applied Psychology (IAAP), and President of the Division of Aging Psychology in the CPS and the China Society for Gerontology and Geriatrics. His research is focused on the mental health of the elderly and on cognitive aging. His publications primarily cover areas of healthy development, mental health, and religious faith.

Satoshi Hirata is Professor at the Wildlife Research Center of Kyoto University, Japan. He has been conducting research on chimpanzees and other great apes from a comparative cognitive perspective to better understand the evolutionary origins of human behavior and cognition. He is currently Director of the Kumamoto Sanctuary of Kyoto University, where ex‐biomedical chimpanzees are housed.

Yuen Wan Ho is a postdoctoral fellow working in the Department of Psychology at the Chinese University of Hong Kong, where she received her PhD degree. Her research interests include personality, aging, and emotion. In particular, she studies how personality and motivational factors could contribute to age differences in emotion regulation and well‐being across cultures.

Etsuko Hoshino is Professor of Psychology at the Faculty of Music, Ueno Gakuen University, Tokyo, Japan. Her research is aimed at understanding relations between music structures and musical affect and she is also actively interested in the influence of background music upon learning contexts, and in music therapy. Hoshino is currently chief editor of the Journal of Music Perception and Cognition (the journal of the Japanese Society of Music Perception and Cognition).

Hiroshi Ishiguro received a D. Eng. in systems engineering from Osaka University, Japan, in 1991. He is currently Professor of Department of Systems Innovation in the Graduate School of Engineering Science at Osaka University (2009–) and Distinguished Professor of Osaka University (2017–). He is also visiting Director (2014–) (group leader: 2002–2013) of Hiroshi Ishiguro Laboratories at the Advanced Telecommunications Research Institute and an ATR fellow. His research interests include sensor networks, interactive robotics, and android science.

Osamu Kitayama is Professor Emeritus of the Department of Clinical Psychology and Community Studies, Kyushu University, Japan. He is a training and supervising analyst, and President of the Japan Psychoanalytic Society. He is author of more than 100 articles, including publications in the International Journal of Psycho‐Analysis in English, and about 15 books on psychoanalysis and medical communication.

Christopher Klager is a doctoral student and University Distinguished Fellow in the Education Policy program at Michigan State University, USA. His research focuses on developing students’ career interest in STEM and STEM teaching. Currently he works on the Crafting Engagement in Science Environments (CESE) project, investigating how to make high school chemistry and physics classes more engaging for students.

Cara Laney is an Associate Professor at the College of Idaho in Caldwell, Idaho, USA. Her research interests include false memory, eyewitness memory, and emotion. She has published more than 30 peer‐reviewed articles and book chapters.

Goh Chee Leong is Dean of the Faculty of Behavioural Science at HELP University, Malaysia. He is former President of ARUPS (ASEAN Regional Union of Psychological Societies) and the Malaysian Psychology Association (PSIMA), and has served as consultant for many organizations, including UNICEF, Maxis, Petronas, DiGi, and CIMB. His research interests include work psychology, stress, and eyewitness memory.

Choong Li Li is presently lecturer at the Department of Psychology at HELP University, Malaysia. Her research interests are in individual and family counseling, with a particular focus on non‐substance addiction such as gambling, video, or online gambling. She is actively involved in voluntary work with orphanages, schools for children with special education needs, and old folks’ homes.

Elizabeth F. Loftus is Distinguished Professor of Psychology and Social Behavior and Criminology, Law, and Society, and Professor of Law and Cognitive Science at the University of California, Irvine, USA. Loftus’s research for the last 40 years has focused on the malleability of human memory. She has been recognized for this research with seven honorary doctorates and election to the National Academy of Sciences, the American Philosophical Society, and the Royal Society of Edinburgh. She is past President of the Association for Psychological Science, the Western Psychological Association, and the American Psychology‐Law Society.

Masako Myowa is Professor at the Graduate School of Education of Kyoto University, Japan. Her research interests include the emergence and development of human intelligence and its evolutionary foundations. In her work, she has taken the approach of comparative cognitive developmental science, comparing the development of cognition in humans and nonhuman primates from their prenatal periods.

Seiichiro Namba is Emeritus Professor at Osaka University, Japan. He is also a member of the Japan Academy. His main area of research is the psychology of hearing. He has served as President of the Acoustical Society of Japan and President of the Japanese Society for Music Perception and Cognition. He received Doctor of Philosophy honoris causa from Oldenburg University, Germany, in 1996, and Commendations for Contributions in Environmental Conservation from the Minister of the Environment in 2003.

Carol D. Ryff, PhD is Director of the Institute on Aging and Hilldale Professor of Psychology at the University of Wisconsin–Madison, USA. Her research centers on the study of psychological well‐being, an area in which she has developed multidimensional assessment scales that have been translated to more than 30 different languages and are used in research across diverse scientific fields. Her research has addressed how psychological well‐being varies by age, gender, socioeconomic status, ethnic/minority status, and cultural context as well as by the experiences, challenges, and transitions individuals confront as they age. This work has generated over 200 publications. She currently directs the MIDUS (Midlife in the US) longitudinal study.

Barbara Schneider is the John A. Hannah Chair University Distinguished Professor in the College of Education and Department of Sociology at Michigan State University, USA. She has used a sociological lens to understand societal conditions and interpersonal interactions that create norms and values that enhance human and social capital for the past 30 years. Her research focuses on how the social contexts of schools and families influence the academic and social well‐being of adolescents as they move into adulthood. She has published 15 books and over 100 refereed journal articles that focus on the family, social context of schooling, and sociology of knowledge.

Rainer K. Silbereisen Professor Emeritus since early 2017, is former Chair of Developmental Psychology and Director of the Center for Applied Developmental Science at the University of Jena, Germany. His main research areas are lifespan human development, with a strong emphasis on the interaction of personality with ecological conditions, such as cultural contexts and immigration, and rapid social, economic, and political change.

Tracy L. Spinrad is Professor of Family Studies in the T. Denny Sanford School of Social and Family Dynamics at Arizona State University, USA. Her program of research focuses on the socioemotional development of young children, particularly the relations of children’s self‐regulation abilities (i.e., effortful control) to children’s social adjustment. Further, much of her work has examined the role that parenting plays in the development of young children’s moral development, altruism, and empathy.

Anwarul Hasan Sufi is Professor of Psychology of the University of Rajshahi, Bangladesh, and Director of the Rajshahi University Mental Health Center. Besides his specialization in developmental disabilities, his research interests are in aviation psychology and clinical psychology. He has written books in English and Bengali in the field of psychology, and is co‐author of textbooks on basic psychology for college students in Bangladesh. He has served as consultant for national and international NGOs working in Bangladesh in the areas of health, education, and disabilities and has been Guest Professor at universities in North America, Europe, and Asia.

Eugene Y. J. Tee is Senior Lecturer at the Department of Psychology, HELP University, Malaysia. He attained his PhD in Management from the University of Queensland in 2010 and has research interests in the study of emotions‐related processes in social and organizational interactions. He has published work on emotions in leader–follower interaction in Leadership Quarterly, Advancing Relational Leadership Theory, and Research on Emotions in Organizations.

Masayoshi Tsuge is Professor in the Faculty of Human Sciences at the University of Tsukuba, Japan. His main area of research is on intellectual disability, developmental disabilities, and behavioral disorders, with a special focus on special needs education.

Carlos Valiente is Professor at Arizona State University, USA. He studies the development of children’s emotional, social, and academic functioning and is especially interested in understanding when and why emotion and self‐control are related to success in the academic domain. His main research projects involve a longitudinal study designed to examine the role of classmates’ temperament on children’s academic functioning and a twin study that aims to explicate genetic and environmental mechanisms associated with sleep and health behaviors.

Jingjing Wang is a PhD student at the Chinese Academy of Sciences (CAS) Key Lab of Mental Health at the Institute of Psychology, and the University of CAS, Beijing, China. Her main area of research is on the cognition, emotion, and mental health of older adults.

Stuart K. Watson is currently a Research Fellow at the University of Zurich, Switzerland. His research interests span social learning, communication, and cultural transmission in primates and birds.

Andrew Whiten is Wardlaw Professor of Evolutionary and Developmental Psychology at the University of St. Andrews, UK. His research interests focus on the evolution and development of social cognition, particularly social learning and culture in human and nonhuman primates.

Lindsey Young is a first‐year doctoral student, Rasmussen Fellow, and Erickson Research Fellow in the Education Policy program at Michigan State University, USA. Her research interests include science curriculum development and evaluation.

Liyu Zhan is Associate Professor of Psychology and Deputy Director of the Mental Health and Guidance Center of Fujian Agriculture and Forestry University, China. She has received awards as an outstanding psychological educator at the university. Her main area of research is the mental health of college students. She has been a visiting scholar at the Huizhen Ke Lab (Asia University, Taiwan), working on suicide intervention and problematic Internet use. She has published several papers in Chinese journals.

Preface

This edition of the Proceedings of the International Congress of Psychology comprises highlights from one of the most successful international psychology conferences since the beginning of the twenty‐first century. In July 2016, over 8,000 attendees – most from outside of Japan – met in Yokohama to participate in 7,800 presentations and sessions that were offered in many formats. Beyond size, the Congress was also highly successful in terms of scope and quality, offering a wide‐ranging program that covered the most recent developments in all areas of psychology.

The contents of this Proceedings book have been selected to reflect the ICP 2016 Congress theme of Diversity in Harmony: Insights from Psychology. The editors invited all who delivered keynote addresses to contribute and selected for inclusion some addresses and presentations from the invited symposium and open lecture series. In this way, the Proceedings book is intended to offer a collection of interesting and stimulating readings rather than a set of refereed research papers.

While the field of psychology is often divided into a number of area specializations, the actual content of research may not easily be classified into a single category. Often research breakthroughs involve perspectives and methodologies encompassing a multiplicity of disciplinary areas. For example, psychology today explores the human mind in the prehistoric era, compares the minds of primates and contemporary humans, and examines human cognitive capability using Artificial Intelligence (AI). Recently, a number of books about the entire history of Homo sapiens have gained a wide readership. In examining human interaction with others – ranging from humans and primates to bacteria, for example – findings persuasively conclude that human beings are the result of complex evolution over a very long time, and that studying the past is essential to understanding the mechanisms and systems of the contemporary human mind. Contrastingly, human capabilities are being rapidly expanded through progress in AI and it is already clear that AI technology will inevitably change many aspects of human life. Primatology and AI are just two examples of psychology’s growing collaborative work with neighboring fields.

The Proceedings book is divided into two parts. In Part I, Psychology Approaching New Synergies, we have included research that offers recent exciting new insights gained from multidisciplinary perspectives and methodologies. In Part II, as the title Psychology Confronting Societal Challenges suggests, we have included chapters that put psychology – as the study of the human mind – at the center of our understanding and ability to address the many problems facing groups and individuals in modern society. As the chapters included in this section show, the social problems identified always involve the human factor, but are complex and often require multidisciplinary approaches. Of course, psychology continues to be useful in addressing individual problems.

Taken as a whole, the content of the Proceedings book is a reflection of the state of psychology in the second decade of the twenty‐first century and it would seem there is much about which to be optimistic. Two important characteristics are especially evident: multidisciplinary approaches are increasingly taking advantage of technological advances, and contributions from researchers and practitioners from regions beyond Europe and North America are expanding. The editors of this Proceedings book are both impressed by current achievements in the field and encouraged by the promise of even greater progress to come.

Acknowledgments

The Editors would like to thank all those who have worked so hard to bring this book into being. First, as the title states, the contents are a reflection of the International Congress of Psychology (ICP) 2016, held in Yokohama, Japan, and the editors want to express their sincere gratitude to all those who contributed so much to making it such a highly successful Congress. Here we would like especially to acknowledge the invaluable help of the office of the Japanese Psychological Association and of members of the ICP 2016 executive committee, namely: Toshikazu Hasegawa (secretary‐general after May 2016 and chair of fund raising), Masataka Watanabe (secretary‐general until April 2016), Yuji Hakoda (vice chair of scientific program), Makiko Naka (co‐vice chair of scientific program and chair of emergent psychologist/scholar program), Tatsuya Kameda (co‐vice chair of scientific program and vice chair of general affairs), Kaori Karasawa (co‐vice chair of general affairs), Toshihiko Hinobayashi (chair of finance), Jiro Gyoba (chair of publicity), Akiyoshi Kitaoka (co‐vice chair of publicity), Kazuhisa Takemura (co‐vice chair of publicity), Kyoko Noguchi (chair of local host), Kiyoshi Ando (co‐vice chair of local host), Koji Takenaka (co‐vice chair of local host), Masuo Koyasu (co‐vice chair of local host), and Atsuko Suzuki (Japanese Psychological Association liaison). Thanks must also go to the International Union of Psychological Science (IUPsyS), under whose auspices the Congress was held, especially to the Officers of IUPsyS and to Rainer K. Silbereisen as IUPsyS/ICP liaison for his continued support.

In terms of the book itself, our thanks go to the authors for their contributions that offer readers such a tremendous insight into the diverse and interesting world of psychology. We are truly grateful for their efforts in support of the ICP and this publication.

With regard to the actual book production, special thanks are due to Verona Christmas‐Best, who took on the role of managing editor and efficiently handled the final stages of bringing the book together and the many associated editing obligations. Finally, thanks must go to our publishers, Wiley, for their supportive, generous, and sympathetic handling of this project.

Diversity in Harmony – Insights from Psychology

Part I

Psychology Approaching New Synergies

1

Social Cognition, the Amygdala, and Autism

Ralph Adolphs

California Institute of Technology, USA

1.1 Three Broad Themes

At the outset, there are three broad themes that are important to consider that will guide the rest of this chapter. These are that (1) social cognition has enabling, or antecedent conditions; (2) the social world is complex; and (3) any specific method has fundamental limitations. We will discuss all these points with a focus on face processing, and through examples of findings in a psychiatric disease, autism, and in cognitive neuroscience, with a focus on the amygdala. Each of these three points suggests important ways forward, which we will discuss in further detail.

To help frame the discussion, we begin with a brief introduction to autism; we discuss the amygdala further below. Autism is a psychiatric disorder recognized since the 1940s, when Kanner and Asperger contemporaneously identified the disease in children (Kanner, 1943). It is a disease that arises early in life, and remains pervasive throughout life. Although it is currently diagnosed around age 3, there are precursors to it that already predict whether a child will develop autism or not. Autism is highly heritable, although no single gene accounts for a large percentage of autism; instead the disease arises from polymorphisms across many genes, each typically contributing only a very small effect size in isolation. These genes in turn code for protein products that influence many aspects of brain development and function, and in particular aspects of how neurons make and maintain synaptic connections with one another. Abnormal connectivity in the brain is currently one leading hypothesis for an intermediate phenotype that accounts for a substantial fraction of autism (Geschwind & Levitt, 2007). This abnormal connectivity in turn causes abnormal brain function that manifests as a particular profile of abilities and disabilities – the ones used to diagnose the disease, which currently can be diagnosed only on behavioral criteria, not by a medical or genetic test of some kind.

In the psychiatric reference book used to diagnose disorders (the Diagnostic and Statistical Manual, DSM), autism was diagnosed as featuring impairments in three domains: social interaction, language, and stereotyped and repetitive behaviors. The first two are related, and have become fused in the transition from DSM‐IV to DSM‐V. The third is a somewhat heterogeneous category of impairments, including not only repetitive behaviors but also rigidity, and exceptional focus and attention to highly specific objects or topics. It has long been recognized that autism is a spectrum, and so it is often referred to as autism spectrum disorder (ASD), and it covers a very wide range from high‐functioning individuals who have PhDs and whose primary complaint is skill in social interactions, to low‐functioning individuals who are mentally retarded and mute. It remains an open question of considerable interest whether the processing deficits and behaviors seen in autism are truly continuous with the psychiatrically healthy population, and whether there might be subtypes of autism. It is hoped that research on the themes described below could help to answer these questions.

1.1.1 Antecedent Causes to Social Cognition

The first theme, that social cognition has antecedent conditions, is fairly obvious once we think about it. Social cognition does not emerge out of nowhere. It develops; it is caused by other processes; and it requires embedding in many other psychological processes in order to generate cognition and social behavior.

Perhaps the two most investigated antecedent conditions for social cognition are attention and motivation. Attention has long been noted to be critical for filtering sensory information, and could thus be thought of simply as a filter that determines sensory inputs, on which subsequent social cognition might be based. Thus, if we pay attention to somebody’s face, we are able to make judgments about the emotion expressed on the face. If we do not pay attention to the face, we are unable to make such judgments (or make them much more poorly). But attention is much more than merely a filter, and should probably be thought of as an active seeking out of socially relevant information. We explore the visual world with our eyes, for instance, sampling relevant features as we make fixations onto them. Indeed, eyetracking has often been used to measure (overt) visual attention. This more active, instrumental view of attention of course raises a next question: so how do we decide where to attend in the first place? Presumably the value, salience, and interest of particular features of stimuli motivate us to pay attention to them. Thus, motivation is another key antecedent process that guides social cognition, together with attention.

Motivation can be thought of simply as that which causes instrumental behavior. Insofar as visual attention can be thought of as instrumental behavior, motivation can cause visual attention. An example would be top‐down visual search, as when we are trying to find a person in a crowd. Conversely, it is also likely that attention influences motivation, since it is well known that our attention to stimuli influences both our preferences and choices. This particular association has been quantified with models such as drift‐diffusion models, which model the accumulation of evidence that can cause motivation and choice. For instance, the more we look at a particular face, the more we are inclined to choose it as the preferred one, in two‐alternative choice tasks with similar faces (Shimojo, Simion, Shimojo, & Scheier, 2003).

There are several specific factors that have been identified that contribute to motivation, and hence to attention. Perhaps the clearest one, and the one best studied in the laboratory, is reward value. If we find a particular feature rewarding, or predictive of reward, we will be motivated to attend there. This would be the simplest kind of explanation to account for why we like to thumb through magazines that have lots of pictures of people: images of people are intrinsically rewarding, and our attention is captured by them. But there are also other factors that can influence attention and motivation: attention can be captured by low‐level saliency, such as the distinctiveness of a stimulus, and this in turn can drive motivation. We are also motivated to seek out information, even when it is not yet known whether that would lead to reward, and even when it is not distinctive. Reward value, saliency, and information are thus at least three factors that could in turn drive attention and motivation (Gottlieb, Hayhoe, Hikosaka, & Rangel, 2014), which in turn drive social behavior.

There is evidence to support the operation of all three factors with regard to face processing. Faces and other visual social stimuli are rewarding (Deaner, Khera, & Platt, 2005), and this rewarding property just of images of faces seems to be diminished (relative to other rewards, such as money) in people with autism (Lin, Rangel, & Adolphs, 2012). Their saliency is evident from the efficiency with which they can be detected in visual search, again an aspect that is impaired in people with autism (Wang et al., 2014), although the impairment in autism appears to be broader than just for faces (Wang et al., 2015). Finally, the information content of regions of the face drives how we attend to those regions. An interesting cross‐cultural finding is that Asian observers tend to look more at the eyes in faces and less at the mouth than do Caucasian observers. A presumptive explanation for this is that the mouth carries less information in Asian people, because of cultural display rules that lead to reduced emotional expression around the mouth (Caldara, 2017).

Motivation and attention to social stimuli are thought to be dysfunctional in autism. One highly influential hypothesis about autism proposes that infants and children with autism do not find social stimuli (other people, faces) rewarding, and so are not motivated to attend to them (Chevallier, Kohls, Troiani, Brodkin, & Schultz, 2012). The developmental consequence of this deficit could then translate into social cognition difficulties later in life: if you do not attend to faces, you will not process faces as often, and consequently your brain will not develop expertise with faces, as it does in typically developing individuals. There is recent evidence that coarse mechanisms for attending to face‐like configurations of visual stimuli may be present already in the womb: fetuses orient preferentially to lights in the configuration of eyes and mouth, when these are projected onto the abdomen of the mother (bright light can penetrate into the womb) (Reid et al., 2017).

There is a final important point to make. It is usually assumed that motivation and attention are domain‐general processes that come into play at the front‐end, so to speak, and that the apparent domain specificity of social cognition arises from subsequent mechanisms. But as we noted, motivation and attention can themselves exhibit selectivity for certain stimuli or features, and so can play a role both in the contemporaneous selective processing of social stimuli and in the development of domain‐specific processes through experience (Spunt & Adolphs, 2017). It is even possible that attentional and motivational processes are sufficient to produce apparent category selectivity, if they amount to an intelligent enough filtering mechanism. For instance, if one combined attention to certain coarse features (the triangular configuration of eyes and mouth), and certain statistically specified locations in space (e.g., usually in the upper visual field, or foveal), and certain conjunctions of context (e.g., faces and voices), cells responding to such simple cues and their conjunctions might, in the aggregate, result in selective processing of faces.

1.1.2 The Social World is Complex

The second theme mentioned above is that realistic social stimuli are inherently complex. Other people, if we consider them as stimuli for a moment, are multimodal, moving objects with many features and attributes that all need to be processed together. They also occur in context, often involve substantial memory, and engage cognitive and behavioral processes that are typically bidirectionally interactive. Even just an image of an isolated face is complex, which is why it has been difficult to design computer vision algorithms to recognize faces. Many different features, and their relationships amongst one another, need to be represented in a flexible, viewpoint‐invariant way, and need to be linked rapidly to the retrieval of often large amounts of semantic knowledge about the person whose face we are seeing.

The inherent complexity of social stimuli has typically been dealt with in the laboratory by using vastly impoverished stimuli, since these are easier to analyze and control. However, this is no longer necessary, since it is possible now to collect large amounts of data quickly, and to construct computational models that analyze such data. Some examples of this will be presented below, but it is actually a rather common emerging theme in social neuroscience (see Adolphs, Nummenmaa, Todorov, & Haxby, 2016).

One way that the brain deals with the complexity of social stimuli is by representing them in a space with much lower dimensions. The identity of familiar individuals may be represented efficiently in a space with perhaps as few as 50 dimensions, and can be decoded from small ensembles of neurons, at least in experiments with monkeys (Chang & Tsao, 2017). More relevantly here, the social attributions that we make about people from their faces – their intentions, emotions, potential threat, and so forth – are likely represented in a space with only a few dimensions. Psychologists who study the impressions we glean from faces have identified three broad dimensions that account for much of variance in our attributions: attractiveness, dominance, and valence (or trustworthiness). There is considerable consensus, at least within a given culture, in the social attributions that we make from faces, and we are able to make them surprisingly rapidly, with less than 100 ms viewing time. Many of the core attributions are already seen in infants. It is an intriguing and very important general fact that we tend to be much more confident of our social attributions than we ought to be: we make the social judgments automatically and quickly, but they reflect more of our biases and stereotypes than providing accuracy. Alex Todorov’s book, Face Value, provides a nice review of these effects (Todorov, 2017).

Two final sources of complexity are context and interaction. The social judgments that we make about other people depend critically on context, and in the real world involve interactions. Studying this dynamic and situated aspect of social cognition has been difficult and typically overlooked (Przyrembel, Smallwood, Pauen, & Singer, 2012), but there is now considerable interest in interactive experimental protocols, some with face‐to‐face encounters between people, others using virtual reality. These will be important directions for future development, conceptually, methodologically, and also in terms of the analysis tools.

1.1.3 Comparing Between Methods

The third and final broad theme of this chapter is that one must make comparisons across multiple approaches. No single approach will suffice, since each approach has limitations and shortcomings. Thus, the strongest eventual syntheses will come from studies that combine methods, or even species. Examples would be studies that use the same stimuli, and ask the same question, with electrophysiology and fMRI; or that ask parallel questions in monkeys and in humans; or that use correlational methods like fMRI as well as more causal methods like TMS or lesions. Of course, achieving this in a paper from a single laboratory is typically impossible. This highlights the need for collaborations as well. Ultimately, we want social neuroscience to be a cumulative science in which multiple data points can build toward a convincing story, not isolated snippets that are difficult to compare.

It is worth briefly noting the major limitations with some of the most popular methods. It is well known that functional neuroimaging has clear advantages and disadvantages, for instance. Its strengths are its noninvasive nature, and whole‐brain field‐of‐view. Limitations are the typically very small effect sizes and indirect nature of the primary measure (changes in magnetic susceptibility due to changes in blood oxygenation), artificial environment, modest spatio‐temporal resolution, and correlational nature of the conclusions that are obtained (although there are methods that involve causal modeling as well). While electrophysiological measures such as EEG have similar limitations (but much better temporal resolution), invasive intracranial recordings in surgical patients provide the best spatio‐temporal resolution – we give an example at the end of this chapter. Yet all these measures are primarily correlational in nature (although causal inferences can be derived from them with some effort), emphasizing the importance of perturbative approaches, such as TMS or lesion studies (which we also review below). The most compelling conclusions are ones that can be drawn from multiple approaches.

1.2 Impaired Attention to Eyes in Faces Following Human Amygdala Lesions

The example set of studies from our laboratory emphasize the first of the above three broad themes: the critical role of attention in social cognition. The story is particularly relevant, because it shows how an initially rather complex‐seeming, and unexplained, specific deficit in one aspect of social perception (an inability to recognize fear in faces) could actually be explained, and even experimentally cured, through understanding attention.

The story begins with a famous patient, a woman named S.M., whom we have studied over several decades and who has provided the field of affective neuroscience with a wealth of insights about the necessary role of the amygdala in human social cognition and behavior (see Feinstein, Adolphs, & Tranel, 2016 for review) (Figure 1.1). S.M. has Urbach‐Wiethe syndrome, an extremely rare genetic disease that results from deletions or mutations in the gene coding for extracellular matrix protein 1, a structural protein that is expressed not only in the brain but in many other organs. This disease, for reasons unknown, causes calcifications and lesions in the medial temporal lobe in a subset of patients (Hamada et al., 2002; Hofer, 1973). In S.M.’s case, it resulted in very focal and complete lesions of the amygdala, on both sides of the brain. The consequences of this in S.M.’s life have been profound: she does not seem to experience fear at all, and thus exhibits behaviors that have often put her at extreme risk (Feinstein, Adolphs, Damasio, & Tranel, 2011).

Figure 1.1 The brain and face processing in patient S.M. Bilateral amygdala lesions impair the use of the eyes and gaze to the eyes during emotion judgment. (A) A patient with bilateral damage to the amygdala made significantly less use of information from the eye region of faces when judging emotion. (B) While looking at whole faces, the patient (right column of images) exhibited abnormal face gaze, making far fewer fixations to the eyes than did controls (left column of images). This was observed across emotions (free viewing, emotion judgment, gender discrimination). (C) MRI scan of the patient’s brain, whose lesion was relatively restricted to the entire amygdala, a very rare lesion in humans. The two round black regions near the top middle of the image are the lesioned amygdalae. (D) When the subject was instructed to look at the eyes (SM eyes) in a whole face, she could do this, resulting in a remarkable recovery in ability to recognize the facial expression of fear. The findings show that an apparent role for the amygdala in processing fearful facial expressions is in fact more abstract, and involves the detection and attentional direction onto features that are socially informative.

Source: © Ralph Adolphs.

It is important to say a few words about the amygdala and the human lesion cases here. The amygdala has long been implicated in fear, and there is substantial evidence across animal species including humans that it is necessary for many aspects of fear processing (Amaral & Adolphs, 2016), even though its role in the conscious experience of fear remains debated, especially in animals (LeDoux, 2017). Two limitations of lesion studies of the amygdala are noteworthy. First, as with all lesion studies, the loss of function observed in a lesion case does not warrant the conclusion that the lesioned structure normally causes the function. So although amygdala lesions impair many aspects of fear processing, this doesn’t mean that the amygdala normally implements those aspects of fear processing. Luckily, there is considerable evidence from other approaches that indeed does support that conclusion. Second, the amygdala is a complex structure consisting of a dozen different nuclei with further intermingled cell populations that subserve different functions. Lesions and fMRI thus have inadequate spatial resolution to resolve these populations, an issue that requires techniques like optogenetics, in which genetically targeted populations of cells can be manipulated. Plenty of those optogenetic studies have also been done now, and while they provide much more detail to the story, they largely support the conclusion that the amygdala participates in processing threat‐related stimuli, although it also participates in processing rewarding stimuli. The modern‐day conclusions are thus that the amygdala contains cell populations that implement functions that contribute to fear and anxiety. It also contains other cell populations that have different functions, and there are also other structures in the brain that participate in processing fear and anxiety. Whether a unitary function of some kind can be ascribed to the amygdala remains unclear, but when this has been attempted, functions related to social cognition have almost always emerged (Adolphs, 2010; Rutishauser, Mamelak, & Adolphs, 2015).

Across a large number of experiments, it was found that S.M. is selectively impaired in her ability to recognize fear from facial expressions. Although her basic vision is normal, and although she can discriminate all faces, even fear faces, normally, she fails to be able to recognize that a facial expression of fear signals the emotion fear (Adolphs, Tranel, Damasio, & Damasio, 1994). This deficit was subsequently discovered to be correlated with an inability to make use of the eye region of faces (Adolphs et al., 2005). To show this, we used a technique called bubbles in which participants were shown small, random pieces of a whole face and asked to recognize the emotion. Such a task, across many trials, can give us a classification image that shows which regions of the face carry discriminative information that allows viewers to classify them as fear or another emotion. In S.M.’s case, she had a very specific impairment on this task: she failed to make use of information from the eye region of the face. This made a lot of sense, since the eye region is normally the region of the face that is most informative about fear: wide eyes signal fear (Smith, Cottrell, Gosselin, & Schyns, 2005). So an inability to use this information from the eyes should result in impaired fear recognition, providing a mechanistic explanation for why S.M. was impaired in recognizing fear.

This finding still left two possible hypotheses. One hypothesis would state that S.M. looks at people’s faces normally, and so has available at the level of the retina exactly the same information that healthy people do when she looks at fear faces. Her impaired ability to utilize information from the eye region of faces in order to recognize fear, then, would be traced to a mechanism that depends on the amygdala. The amygdala would be necessary for some further processes that allow the brain to know that wide eyes signal fear.

A second hypothesis, however, would be that S.M. does not even look normally at the face stimuli in our experiment. That is, she might fixate faces in unusual ways, and thus might indeed not have available, at the level of the retina, the same information that healthy individuals do when they look at faces. To distinguish between these two possibilities, we used eyetracking to measure how S.M. looks at faces.

We found that S.M. indeed does not fixate faces normally. Often, she simply stares at the center of the image, not exploring it with her eyes. When she does move her eyes, she does not preferentially look at the eyes in faces, unlike healthy individuals. This finding thus provides a compelling mechanistic explanation of why S.M. is impaired in recognizing fear in faces. Normally, people look at the eyes in our face stimuli, and wide eyes signal fear. However, S.M. fails to look at the eyes in faces, and thus is unable to use information from the eye region of the face to tell her that the face expresses fear.

This story is particularly nice because it makes some testable further predictions. If true, it should be possible to help S.M. to recognize fear in faces. We could simply instruct her to look at faces the way that healthy people look at faces: fixate the eyes in faces. Would this improve her impaired fear recognition? When we did the experiment, we indeed found that it did. Unfortunately, the improvement only lasted the duration of the experiment. Without an explicit instruction to fixate the eyes in faces, S.M. would always revert back to not fixating the eyes, and to showing impaired recognition of fear.

This set of studies thus illustrates the important role of attention in social perception. It also raises the question whether we might find similar results in some other clinical populations that have difficulties in social cognition. One such population are people with autism, who also report difficulties figuring out how other people feel, and who are also often described as making poor eye contact. We turn to this clinical population next.

1.3 Atypical Visual Attention in People with Autism

The two antecedent processes that we mentioned as enabling social cognition have both been reported to be impaired in people with autism, and according to some hypotheses are thought to be responsible for the development of social difficulties in autism (Chevallier et al., 2012). It is known that people with autism fixate faces in unusual ways (Pelphrey et al., 2002) and it has also been reported that people with autism do not find pictures of faces normally rewarding in guiding their instrumental behavior (Lin et al., 2012). While there is the belief that these deficits in social attention and social reward are specific, or at least disproportionate, for social stimuli, establishing this specificity is still an important and open question. It is possible that there are broader deficits in attention and reward processing, for all stimuli, and it is also possible that the deficits are specific to certain domains or features of stimuli, or computations performed on them, that happen to be disproportionately important when we process faces.

Be that as it may, the unusual fixation patterns of people with autism onto faces bear some intriguing resemblance to those seen in the patient with amygdala lesions, S.M. (Figure 1.2). Like S.M., people with autism tend to look less at the eyes in faces. While the patterns are far from identical, and while there are many other differences between patients with amygdala lesions and patients with autism, this superficial similarity is one piece of support for the hypothesis that amygdala dysfunction might contribute to autism (Baron‐Cohen et al., 2000). Indeed, there is now overwhelming evidence that the amygdala is abnormal in autism (mostly from histological and structural studies), but it is also clear that (a) many other brain regions are also abnormal in autism, and (b) abnormalities in the amygdala contribute to all developmental disorders (and probably many adult‐onset disorders), and not just to autism (Schumann, Bauman, & Amaral, 2011). Some of the most detailed ongoing studies that examine amygdala function in autism are using large data sets (such as those from the ABIDE network; Di Martino et al., 2014) to examine functional connectivity of the amygdala with other brain structures (often from resting‐state fMRI data). It may be possible to diagnose autism just from the pattern of resting‐state functional brain activation, although currently the number of false positives with such approaches is still too high.

Figure 1.2 Fixations onto faces in S.M. and in people with autism show similarities. The images show data obtained from how participants fixate features from faces; hot colors denote higher density of fixations (except in the control–autism difference image, where red colors indicate that controls fixate more than autism at that location, and blue colors indicate that people with autism fixate there more than controls). Note that the images for autism and controls are obtained from groups of participants, whereas the image from S.M. is from a single individual.

Source: Ralph Adolphs.

A further investigation by us of visual attention in autism illustrates the second of the broad themes we had outlined at the beginning of this chapter. That is the theme that social stimuli are complex, but with careful characterization can still be analyzed with sophisticated models and sufficient data. We asked the question: What features in visual stimuli capture people’s visual attention, and how might this differ in people with autism? You could think of the answer to this question as producing something like a fingerprint that shows a profile of the weight that each visual feature has in attracting your visual attention.

In our study (Wang et al., 2015), we showed participants many different visual images. Importantly, all of these were natural scenes, and so were quite complex. They contained many different objects and features: people, animals, objects, trees, sky, background, and so forth. To capture all these different features in a computational model, we first used an automated algorithm to determine low‐level visual saliency of specific regions on the image. This algorithm, developed by Christof Koch and Laurent Itti (Itti & Koch, 1998), essentially finds, in an automated way, regions that will attract visual attention because they are bright, or have high contrast, or have a particular color. So this aspect of visual saliency is relatively easy to quantify on our stimuli.

But we also wished to quantify semantic, object‐based features in all our stimuli. You do not only fixate onto a region because it is bright, but also because of its meaning: whether it is showing a face, or an interesting animal, or something emotional. To characterize these semantic‐level properties, we had a large number of students annotate the images (cf. Xu, Jiang, Wang, Kankanhalli, & Zhao, 2014). This produced a detailed model consisting of pixel‐based (low‐level saliency), object‐based, and semantic‐based features. We could then train this model on a subset of the eyetracking data, and ask how well it was able to predict new eyetracking data. The results of this produce a profile across all the different features, showing us how strong an effect they have on visual attention.

We then carried out exactly this same analysis in people with autism, and asked how their visual attention might be driven by different factors. Indeed, we found that there was a difference. Whereas normal controls show fixations that are driven less and less by pixel‐based saliency over time, and more and more by semantic‐based saliency, people with autism show much less of this effect and keep looking at low‐level saliency regions in the image such as contrast and brightness. Thus, this analysis showed that visual attention in autism is characterized, at least in part, by an inability to be guided by the semantic meaning of objects in an image, and instead seems to stick to low‐level cues.

There are many other examples of such a more data‐driven, feature‐based approach that uses relatively complex naturalistic stimuli. One type of stimulus that has become quite popular is videos or movies, which can be shown to subjects while obtaining fMRI data. Not only are these stimuli engaging and thus capable of producing strong brain responses, but they offer a very efficient way of sampling a large range of different features within a context. Decomposing the complex stimulus of the movie into its constituent features is not trivial, but even without detailed decomposition it is possible to use such a rich stimulus to find abnormally activated brain networks in people with autism (Byrge, Dubois, Tyszka, Adolphs, & Kennedy, 2015), and then work backwards from this finding to ask what it is about the movie (e.g., which specific low‐level or semantic‐level features) might be most responsible for this. For instance, it was found that social awkwardness is one time‐varying aspect of the movie that results in abnormal brain activation in autism (Pantelis, Byrge, Tyszka, Adolphs, & Kennedy, 2015).

1.4 Putting it All Together: Single‐neuron Responses in the Amygdala

Finally, we turn to putting all three themes together, and in particular to highlighting the third of the themes, the need to use multiple methods. In this study, we used the bubbles method that was already introduced in Figure 1.1, we recorded from the amygdala, and we investigated amygdala responses in people with autism. The dependent measure this time, however, was not eye movements but single‐neuron responses recorded from depth electrodes in the brains of neurosurgical patients.

These patients are all patients who have medically untreatable epilepsy, and whose seizures cannot be localized adequately with scalp EEG. The clinical goal is to find the region of the brain from which the seizures originate, so that this could be surgically removed. Often, the source is in the medial temporal lobe – in the amygdala or hippocampus – and resecting these tissues in a surgery called a temporal lobectomy can cure the epilepsy. But to decide exactly where the seizure originates, it is essential to be able to record the electrical activity of a seizure from electrodes in the brain, permitting a precise determination. For this clinical reason, neurosurgeons implant depth electrodes into the brains of such patients. They then spend 1–2 weeks in the hospital, with wires connected to the depth electrodes, so that one can record when a seizure occurs. During this time, the patients can also elect to participate in research studies, and one can show them stimuli and record single‐neuron responses in the brain obtained through the depth electrodes. This is a very important and rare source of recordings from single neurons in the human brain, which has resulted in significant contributions in cognitive neuroscience (Fried, Rutishauser, Cerf, & Kreiman, 2014).

In our study (Rutishauser et al., 2013), we asked how single neurons in the amygdala would respond to the features of faces. Are their responses driven more by a specific part of the face, like the nose, or the eyes? To answer this question, we