Automated Measurement of Facial Expression in Infant–Mother Interaction: A Pilot Study

Automated facial measurement using computer vision has the potential to objectively document continuous changes in behavior. To examine emotional expression and communication, we used automated measurements to quantify smile strength, eye constriction, and mouth opening in two 6‐month‐old infant‐mother dyads who each engaged in a face‐to‐face interaction. Automated measurements showed high associations with anatomically based manual coding (concurrent validity); measurements of smiling showed high associations with mean ratings of positive emotion made by naive observers (construct validity). For both infants and mothers, smile strength and eye constriction (the Duchenne marker) were correlated over time, creating a continuous index of smile intensity. Infant and mother smile activity exhibited changing (nonstationary) local patterns of association, suggesting the dyadic repair and dissolution of states of affective synchrony. The study provides insights into the potential and limitations of automated measurement of facial action.     

Infant Smiling Dynamics and Perceived Positive Emotion

To better understand early positive emotional expression, automated software measurements of facial action were supplemented with anatomically based manual coding. These convergent measurements were used to describe the dynamics of infant smiling and predict perceived positive emotional intensity. Over the course of infant smiles, degree of smile strength varied with degree of eye constriction (cheek raising, the Duchenne marker), which varied with degree of mouth opening. In a series of three rating studies, automated measurements of smile strength and mouth opening predicted naïve (undergraduate) observers’ continuous ratings of video clips of smile sequences, as well as naïve and experienced (parent) ratings of positive emotion in still images from the sequences. An a priori measure of smile intensity combining anatomically based manual coding of both smile strength and mouth opening predicted positive emotion ratings of the still images. The findings indicate the potential of automated and fine-grained manual measurements of facial actions to describe the course of emotional expressions over time and to predict perceptions of emotional intensity.     

Testosterone, Smiling, and Facial Appearance

In a study of possible links between testosterone and dominance, 119 men and 114 women provided saliva samples for testosterone assay and posed smiling and not smiling for portrait photographs. Expert judges viewing the photographs found smaller smiles among high than low testosterone men, with less zygomatic major (raising the corners of the mouth) and orbicularis oculi (raising the cheeks and crinkling around the corners of the eyes) muscle activity. Naive judges viewing individual photographs gave higher potency ratings to smiling high testosterone men than smiling low testosterone men. Naive judges viewing photographs grouped into high and low testosterone sets gave higher potency and lower goodness ratings to high than to low testosterone men, regardless of whether they were smiling. Among women, judges found only slight relationships between testosterone and facial appearance. The pattern among men of less smiling with higher testosterone levels fits with research linking testosterone to face-to-face dominance.     

Adult Perceptions of Positive and Negative Infant Emotional Expressions

Adults' perceptions provide information about the emotional meaning of infant facial expressions. This study asks whether similar facial movements influence adult perceptions of emotional intensity in both infant positive (smile) and negative (cry face) facial expressions. Ninety‐five college students rated a series of naturally occurring and digitally edited images of infant facial expressions. Naturally occurring smiles and cry faces involving the co‐occurrence of greater lip movement, mouth opening, and eye constriction, were rated as expressing stronger positive and negative emotion, respectively, than expressions without these 3 features. Ratings of digitally edited expressions indicated that eye constriction contributed to higher ratings of positive emotion in smiles (i.e., in Duchenne smiles) and greater eye constriction contributed to higher ratings of negative emotion in cry faces. Stronger mouth opening contributed to higher ratings of arousal in both smiles and cry faces. These findings indicate a set of similar facial movements are linked to perceptions of greater emotional intensity, whether the movements occur in positive or negative infant emotional expressions. This proposal is discussed with reference to discrete, componential, and dynamic systems theories of emotion.     

Maternal Depressive Symptoms and 6‐Month‐Old Infants' Sensitivity to Facial Expressions

We examined 6‐month‐old infants' abilities to discriminate smiling and frowning from neutral stimuli. In addition, we assessed the relationship between infants' preferences for varying intensities of smiling and frowning facial expressions and their mothers' history of depressive symptoms. Forty‐six infants were presented pairs of facial expressions, and their preferential looking time was recorded. They also participated in a 3‐min interaction with their mothers for which duration of both mother and infant gazing and smiling were coded. Analyses revealed that the infants reliably discriminated between varying intensities of smiling and frowning facial expressions and a paired neutral expression. In addition, infants' preferences for smiling and frowning expressions were related to self‐reports of maternal depressive symptoms experienced since the birth of the infant. Potential implications for social cognitive development are discussed.     

Effects of Dynamic Attributes of Smiles in Human and Synthetic Faces: A Simulated Job Interview Setting

We examined the effects of the temporal quality of smile displays on impressions and decisions made in a simulated job interview. We also investigated whether similar judgments were made in response to synthetic (Study 1) and human facial stimuli (Study 2). Participants viewed short video excerpts of female interviewees exhibiting dynamic authentic smiles, dynamic fake smiles, or neutral expressions, and rated them with respect to a number of attributes. In both studies, perceivers’ judgments and employment decisions were significantly shaped by the temporal quality of smiles, with dynamic authentic smiles generally leading to more favorable job, person, and expression ratings than dynamic fake smiles or neutral expressions. Furthermore, authentically smiling interviewees were judged to be more suitable and were more likely to be short-listed and selected for the job. The findings show a high degree of correspondence in the effects created by synthetic and human facial stimuli, suggesting that temporal features of smiles similarly influence perceivers’ judgments and decisions across the two types of stimulus.

On the meaning and implications of facial prominence

Archer et al. (1983) found that visual depictions (e.g., photographs) of men tend to show more face and less of the body (a characteristic that was termed high face-ism) than visual depictions of women. Furthermore, photographs (of both men and women) high in face-ism elicited more favorable impressions than photographs low in face-ism. The present studies examined possible reasons for sex differences in face-ism and their implications concerning the effects of high and low face-ism on interpersonal perception. Study 1 showed that the greater face-ism in photographs of men was less pronounced when the photographs were taken from periodicals that are oriented toward women's issues. Study 2 showed that photographs high in face-ism received higher ratings on dominance, a dimension related to masculinity, but not on positivity, a dimension related to femininity. This study also indicated that facial expressions provided more information about degrees of positivity while body cues provided more information about dominance and submission. Consistent with these latter results, Study 3 showed that amateur drawings portraying kind or hostile persons showed more of the face while drawings presenting dominant or weak persons showed more of the body. The two phenomena—the relationship of high face-ism with impressions of high dominance and the different types of information transmitted by the face and body—were considered in the discussion.The author would like to express his appreciation to Diana R. Satin and BiancaMaria (Mia) Penati for their assistance with this project.     

The Mystery of the European Smile: A Comparison Based on Individual Photographs Provided by Internet Users

The aim of the study was to analyze cross-cultural differences in preference for smiling among the users of one of the most popular instant messaging sites called Windows Live Messenger in terms of facial expression (smiling vs. non-smiling) on the photographs accompanying their profiles. 2,000 photos from 10 countries were rated by two independent judges. Despite the fact that 20 years have passed since the fall of the Berlin Wall, Internet users from a former Soviet bloc appear to smile less often than those from Western Europe. Also, replicating past research, women irrespective of their nationality smiled more than men.     

Brief Report: Useful Information for Face Perception Is Described with FACS

When we perceive the emotions of other people, we extract much information from the face. The present experiment used FACS (Facial Action Coding System), which is an instrument that measures the magnitude of facial action from a neutral face to a changed, emotional face. Japanese undergraduates judged the emotion in pictures of 66 static Japanese male faces (11 static pictures for each of six basic expressions: happiness, surprise, fear, anger, sadness, and disgust), ranging from neutral faces to maximally expressed emotions. The stimuli had previously been scored with FACS and were presented in random order. A high correlation between the subjects' judgments of facial expressions and the FACS scores was found.     

A Fake Smile Thwarts Cheater Detection

People can discriminate cheaters from cooperators on the basis of negative facial expressions. However, such cheater detection is far from perfect in real-world situations. Therefore, it is possible that cheaters have the ability to disguise negative emotional expressions that signal their uncooperative attitude. To test this possibility, emotional intensity and trustworthiness were evaluated for facial photographs of cheaters and cooperators defined by scores in an economic game. The facial photographs had either posed happy or angry expressions. The angry expressions of cheaters were rated angrier and less trustworthy than those of cooperators. On the other hand, happy expressions of cheaters were higher in emotional intensity but comparable to those of cooperators in trustworthiness. These results suggest that cheater detection based on the processing of negative facial expressions can be thwarted by a posed or fake smile, which cheaters put on with higher intensity than cooperators.     

The Duchenne Smile and Persuasion

We investigated persuasiveness as a social outcome of the ability to produce a deliberate Duchenne smile in a role-play task and of a participant’s use of a Duchenne smile while persuading someone in a live interaction. Participants were tasked with persuading an experimenter to drink a pleasant and unpleasant tasting juice as well as not drink a pleasant and unpleasant juice while being videotaped. Participants’ deliberate Duchenne smiling ability was measured by asking participants to smile while acting out “genuine happiness” and also to mask imagined negative affect with a smile. Smiles in the deliberate Duchenne smiling task and the persuasion task were coded for presence of the Duchenne marker, and naïve viewers of the persuasion task made ratings of how pleasant they thought the juice was. Results showed further evidence that a sizeable minority of people can deliberately produce a Duchenne smile and showed that those with this ability are more persuasive. When persuading to drink the pleasant tasting juice, the correlation between the ability to produce a deliberate Duchenne smile and persuasion was partially due to the use of the Duchenne smile while persuading, but this was not the case with the unpleasant tasting juice. When persuading to drink the unpleasant juice, participants who could deliberately put on the Duchenne smile were more persuasive but their persuasiveness was not the result of using a Duchenne smile during the persuasion task.     

The Stability of Facial Attractiveness: Is It What You’ve Got or What You Do with It?

Physical attractiveness is suggested to be an indicator of biological quality and therefore should be stable. However, transient factors such as gaze direction and facial expression affect facial attractiveness, suggesting it is not. We compared the relative importance of variation between faces with variation within faces due to facial expressions. 128 participants viewed photographs of 14 men and 16 women displaying the six basic facial expressions (anger, disgust, fear, happiness, sadness, surprise) and a neutral expression. Each rater saw each model only once with a randomly chosen expression. The effect of expressions on attractiveness was similar in male and female faces, although several expressions were not significantly different from each other. Identity was 2.2 times as important as emotion in attractiveness for both male and female pictures, suggesting that attractiveness is stable. Since the hard tissues of the face are unchangeable, people may still be able to perceive facial structure whatever expression the face is displaying, and still make attractiveness judgements based on structural cues.     

How the Experience of Emotion is Modulated by Facial Feedback

The facial feedback hypothesis states that facial actions modulate subjective experiences of emotion. Using the voluntary facial action technique, in which the participants react with instruction induced smiles and frowns when exposed to positive and negative emotional pictures and then rate the pleasantness of these stimuli, four questions were addressed in the present study. The results in Experiment 1 demonstrated a feedback effect because participants experienced the stimuli as more pleasant during smiling as compared to when frowning. However, this effect was present only during the critical actions of smiling and frowning, with no remaining effects after 5 min or after 1 day. In Experiment 2, feedback effects were found only when the facial action (smile/frown) was incongruent with the presented emotion (positive/negative), demonstrating attenuating but not enhancing modulation. Finally, no difference in the intensity of produced feedback effect was found between smiling and frowning, and no difference in feedback effect was found between positive and negative emotions. In conclusion, facial feedback appears to occur mainly during actual facial actions, and primarily attenuate ongoing emotional states.     

American-Japanese cultural differences in attributions of personality based on smiles

Several studies have already documented how Americans and Japanese differ in both the expression and perception of facial expressions of emotion in general, and of smiles in particular. These cultural differences can be linked to differences in cultural display and decoding rules (Ekman, 1972; and Buck, 1984, respectively). The existence of these types of rules suggests that people of different cultures may hold different assumptions about social-personality characteristics, on the basis of smiling versus non-smiling faces. We suggest that Americans have come to associate more positive characteristics to smiling faces than do the Japanese. We tested this possibility by presenting American and Japanese judges with smiles or neutral faces (i.e., faces with no muscle movement) depicted by both Caucasian and Japanese male and female posers. The judges made scalar ratings of each face they viewed on four different dimensions. The findings did indicate that Americans and Japanese differed in their judgments, but not on all dimensions.David Matsumoto was supported in part by a research grant from the National Institute of Mental Health (MH 42749-01), and from a Faculty Award for Creativity, Scholarship, and Research from San Francisco State University. We would like to thank Masami Kobayashi, Fazilet Kasri, Deborah Krupp, Bill Roberts, and Michelle Weissman for their aid in our research program on emotion. We would especially like to thank the Editor for her excellent suggestions and help in conceptualizing this research.     

Large-Scale Observational Evidence of Cross-Cultural Differences in Facial Behavior

Self-report studies have found evidence that cultures differ in the display rules they have for facial expressions (i.e., for what is appropriate for different people at different times). However, observational studies of actual patterns of facial behavior have been rare and typically limited to the analysis of dozens of participants from two or three regions. We present the first large-scale evidence of cultural differences in observed facial behavior, including 740,984 participants from 12 countries around the world. We used an Internet-based framework to collect video data of participants in two different settings: in their homes and in market research facilities. Using computer vision algorithms designed for this dataset, we measured smiling and brow furrowing expressions as participants watched television ads. Our results reveal novel findings and provide empirical evidence to support theories about cultural and gender differences in display rules. Participants from more individualist cultures displayed more brow furrowing overall, whereas smiling depended on both culture and setting. Specifically, participants from more individualist countries were more expressive in the facility setting, while participants from more collectivist countries were more expressive in the home setting. Female participants displayed more smiling and less brow furrowing than male participants overall, with the latter difference being more pronounced in more individualist countries. This is the first study to leverage advances in computer science to enable large-scale observational research that would not have been possible using traditional methods.     

The effect of expression sequence and expressor identity on judgments of the intensity of facial expression

Subjects were exposed to a sequence of facial expression photographs to determine whether viewing earlier expressions in a sequence would alter intensity judgments of a final expression in the sequence. Results showed that whether the preceding expressions were shown by the same person who displayed the final expression, or different people, intensity ratings of both sad and happy final expressions were enhanced when preceded by a sequence of contrasting as opposed to similar or identical facial expressions. Results are discussed from the perspective of adaptation-level theory.     

Older Adults’ Trait Impressions of Faces Are Sensitive to Subtle Resemblance to Emotions

Younger adults (YA) attribute emotion-related traits to people whose neutral facial structure resembles an emotion (emotion overgeneralization). The fact that older adults (OA) show deficits in accurately labeling basic emotions suggests that they may be relatively insensitive to variations in the emotion resemblance of neutral expression faces that underlie emotion overgeneralization effects. On the other hand, the fact that OA, like YA, show a ‘pop-out’ effect for anger, more quickly locating an angry than a happy face in a neutral array, suggests that both age groups may be equally sensitive to emotion resemblance. We used computer modeling to assess the degree to which neutral faces objectively resembled emotions and assessed whether that resemblance predicted trait impressions. We found that both OA and YA showed anger and surprise overgeneralization in ratings of danger and naiveté, respectively, with no significant differences in the strength of the effects for the two age groups. These findings suggest that well-documented OA deficits on emotion recognition tasks may be more due to processing demands than to an insensitivity to the social affordances of emotion expressions.     

Facial Mimicry of Spontaneous and Deliberate Duchenne and Non-Duchenne Smiles

Increasing evidence suggests that Duchenne (D) smiles may not only occur as a sign of spontaneous enjoyment, but can also be deliberately posed. The aim of this paper was to investigate whether people mimic spontaneous and deliberate D and non-D smiles to a similar extent. Facial EMG responses were recorded while participants viewed short video-clips of each smile category which they had to judge with respect to valence, arousal, and genuineness. In line with previous research, valence and arousal ratings varied significantly as a function of smile type and elicitation condition. However, differences in facial reactions occurred only for smile type (i.e., D and non-D smiles). The findings have important implications for questions relating to the role of facial mimicry in expression understanding and suggest that mimicry may be essential in discriminating among various meanings of smiles.     

Judging Facial Emotion Expressions in Context: The Influence of Culture and Self-Construal Orientation

We assessed the impact of social context on the judgment of emotional facial expressions as a function of self-construal and decoding rules. German and Greek participants rated spontaneous emotional faces shown either alone or surrounded by other faces with congruent or incongruent facial expressions. Greek participants were higher in interdependence than German participants. In line with cultural decoding rules, Greek participants rated anger expressions less intensely and sad and disgust expressions more intensely. Social context affected the ratings by both groups in different ways. In the more interdependent culture (Greece) participants perceived anger least intensely when the group showed neutral expressions, whereas sadness expressions were rated as most intense in the absence of social context. In the independent culture (Germany) a group context (others expressing anger or happiness) additionally amplified the perception of angry and happy expressions. In line with the notion that these effects are mediated by more holistic processing linked to higher interdependence, this difference disappeared when we controlled for interdependence on the individual level. The findings confirm the usefulness of considering both country level and individual level factors when studying cultural differences.