Not long ago, most people would probably judge how trustworthy you were based entirely on your physical appearance. More specifically, the pseudoscience of physiognomy claimed that a person’s facial expressions could tell you a lot about their personality: were they honest, would you get along with them, are they good at their job, etc. Today, we know that kind of thinking is a dangerous pseudoscience. We also know that looks do play a big role in how we evaluate people, and that these evaluations are often based on cultural stereotypes. Psychology professor Alexander Todorov explains how our biases affect the way we treat people, and by extension, how we think of ourselves in relation to them. Alexander Todorov is the author of Face Value: The Irresistible Influence of First Impressions.

ALEXANDER TODOROV: So physiognomy, or the so-called pseudoscience of reading character from faces, has a very, very long history. The first historical document dates all the way back to the time of Aristotle, but it really got extremely, extremely popular in the 18th and 19th century. 

And for a while there were quite a few studies in the beginning of the 20th century—we're talking the '20s, in the '30s—by psychologists finding very, very little evidence for the accuracy of physiognomic inferences. 

And recently there was, for example, a computer science paper claiming that presumably you can guess whether a person is a criminal or not based on their facial image. And actually the history of identifying the criminal has a very, very long history in physiognomy. 

There's lots of studies showing effects of appearance across different domains. In my own lab more than ten years ago we showed that you can predict electoral success based on judgment from the appearance of politicians. Then there have been many other studies in the legal domains. For example, recent studies show that prisoners who are sentenced to death for the same crime as those we were sentence to a life-sentence without parole, the main difference was the prisoners who were sentenced to death looked less “trustworthy.” 

There's been a lot of evidence in economic games. So for example, economists love the sort of economic game where we have interactions and if I don't know anything about you I have a risky choice whether to invest in you or not. 

But if I invest and you're trustworthy, then we are both better off, but if you're untrustworthy you can take away my money and run away. So it creates this risky situation and many studies suggest that in fact if we think the appearance of the other person is untrustworthy, we tend not to invest in these people. 

Interestingly there was a study that was done in Germany and Switzerland when in fact people played anonymously, and they also did different measurements of their faces, it turns out that these measurements that are supposed to predict untrustworthiness or aggressiveness didn't predict at all how people will behave in this kind of an anonymous economic game. 

The interesting part is that these participants, they were incentivized for accurate responsiveness. And so for every accurate response they were essentially trying to predict what the person did in the situation, they ended up losing money because they were relying on these kinds of appearance. 

But you can see how in a real life situation you can have all kinds of self-fulfilling prophecies. 

So we come and play with you in a game, I decided for some reason that you're untrustworthy so my first move is not a cooperative move. Well, you reciprocate in a similar way. I don't trust you, so why would you trust me? And that kind of leads to a self-fulfilling prophecy. 

So you can create these kinds of dynamic inconsistency in the situation, where in fact the people could be perfectly trustworthy and might cooperate, but the appearance cues can get into the way. 

Now, to the extent that we agree on these impressions an important empirical question is: well, where is this agreement coming from? And so in the past decade we have built mathematical models and other groups have built their own models that essentially are trying to discover the cues that people are using when they form these impressions. 

So what kinds of cues can you observe? Well, when you're deciding whether somebody is trustworthy or not one of the most important inputs is emotional expressions. 

So often in psychology we talk about that there's emotionally neutral faces and there's faces expressing different emotions, but the fact of the matter is that an emotional neutral face is a fiction, is a psychological fiction. You could be emotionally neutral, but if you haven't been sleep deprived you would look better, if you have a nice day you will look better even if you're not smiling. So although the expressions might not to be explicit or exaggerated the way they are typically studied in studies of emotional expressions, they're subtle expressions on our faces. 

So these expressions are a very important input to impressions. Notice they might be accurate, these impressions, at that particular moment here and now, but they would be very lousy as a guide to what the person is in general across time and situations. 

To get back to trustworthiness, faces that look like they're smiling, they have positive expressions, they're perceived more positively. Faces that look disgruntled they're perceived more negatively. What are the other inputs? 

Many of the other inputs come from stereotypes. 

For example, feminine faces are perceived as more trustworthy than masculine faces, on the other hand masculine faces are perceived as more dominant. So there are lots of cues and they are cues that are actually completely idiosyncratic to others. 

For example, faces that resemble people that you already like you would tend to like, not knowing anything about these people. And the other way around faces that resemble people that you dislike you will tend not to like, barring no other information about these people.