Addressing bias in facial recognition

By: Selma O’Malley ’26

Having trouble distinguishing between faces, especially those of a different race from the viewer, might have more to do with psychology than vision. Deja Simon-Jennings, assistant professor of psychology, is working with a team of students to understand the science behind facial recognition.

Prof. Deja Simon-Jennings partnered with psychology majors Charlotte Bowler ’27 and Erin Rapuano ’27 to continue this research over the summer. |Photo by JaQuan Alston

“I study how racial bias influences how we view faces, remember faces, perceive people, and what judgments we make about people,” she says.

Simon-Jennings designed a training to enhance facial recognition and combat the “Other Race Effect,” the human inclination to be worse at distinguishing between faces of other races than our own.

“There are real-world problems that result from this deficit,” Simon-Jennings says, citing eyewitness misidentification in legal proceedings and professors mixing up students of the same race in the classroom.

The training, a 1-hour online practice and questionnaire, is built to improve general face recognition. Across three experiments, 900 participants in the training completed five lessons focused on facial characteristics that are most distinguishing between different races, according to Simon-Jennings: forehead length, inner eye space, nose width, nose to chin length, and mouth width.

“The purpose of the training was trying to draw their attention to these different features and what variability in these features looks like, but most importantly, learning to put it all together,” she says.

Simon-Jennings calls this “holistic processing,” or seeing the whole composite of a face instead of features in isolation, as this is the best way humans distinguish between faces.

With help over the summer from psychology majors and Bergh Family Fellows Charlotte Bowler ’27 and Erin Rapuano ’27, Simon-Jennings will be examining qualitative data from participants’ responses to free-response prompts to see if the training improves holistic processing of faces based on the language used by participants.

“As forehead length increases, how does it affect the appearance of the faces? How do the spatial relationships between elements of each face appear to change?” reads one free-response question from the training.

Prof. Deja Simon-Jennings discusses a training used across three experiments. | Photo by JaQuan Alston

Bowler and Rapuano turned the responses to these questions into a numbered code to label how descriptive and holistic the responses are. This text-to-data method will allow Simon-Jennings to analyze responses and evaluate the effectiveness of the training.

The researchers met weekly to review responses that were independently coded to compare. The goal? An 80% overlap between Bowler and Rapuano’s codings to move forward, which was more difficult to coordinate than it first appeared.

“Someone said, ‘The nose isn’t that big,’ so is it still big?” Bowler says. “You’re saying it’s not that big? Does that mean it’s average? Does that mean it’s small?”

This attention to detail is an important skill for psychology students to practice.

“This level of working and picking up on my small nuances in text, that practice will be very beneficial in future endeavors for them,” Simon-Jennings says.

Bowler says her favorite part about the experiment was the constant progress and when the team was able to meet that coveted 80%.

“It was fun to be able to go through and see our improvements and see what we needed to change, and being able to comfortably work together to make positive improvements,” she says.

Prof. Deja Simon-Jennings hopes to make the training open source, available for anyone to use online. | Photo by JaQuan Alston

Bowler and Rapuano emphasized the collaborative and positive environment that Simon-Jennings cultivated this past summer.

“[Simon-Jennings] is just on the same page as we are, and it feels like I’m working with a peer,” Bowler says.

If the training proves to be effective, Simon-Jennings says she would make it open source, available for anyone to use online.

“There’s some more tweaking and work that needs to be done, but, at the end of the day, I’ve always cared about improving interpersonal interactions, and so I think it’s cool this work could potentially help with that,” she says.