Teaching algorithms about skin tones

When Ellis Monk’s wife became pregnant in 2019, the couple became curious about what skin tone their child might have. The subject was of more than passing interest to the sociology professor, some of whose work involves the role lighter and darker skin tones play in society. Monk’s wife noted that a comprehensive scale would be useful and urged him to develop one.

So he did, and in May Google adopted Monk’s namesake 10-shade scale as a standard in its digital products to make them more inclusive and diverse and to promote wider awareness of the problems and unintended bias associated with technologies that fail to recognize a wider range of skin tones.

Monk’s scale is already making an impact. It has been incorporated in Google’s online image searches and photo filters. The innovation will be particularly important for training artificial intelligence and machine learning applications, such as facial recognition and self-driving vehicle systems, which often have not performed as well with people with darker skin tones. And it could help reduce or eliminate some unintended algorithm bias in search engines and other products.

“She encouraged me to keep it going,” Monk said, crediting his wife. “Eventually, I delved in and put together the Monk Skin Tone Scale, taking the baton [and] building on what she had started.”

Monk’s scale is a rethinking of the Fitzpatrick Scale, which has been considered the online standard. Created by dermatologist Thomas B. Fitzpatrick at Harvard Medical School in 1975, the Fitzpatrick Scale was developed to classify six tones based on skin pigmentation and response to sun exposure. It was not created with diverse populations in mind but as a gauge of tolerance for ultraviolet light, Monk explained.

The Fitzpatrick Scale has been in the open domain for decades and was adopted by the tech industry primarily for use in machine learning and artificial intelligence. Humans must annotate images or videos to train machine-learning algorithms for things like face detections.

“Instead of using this Fitzpatrick Scale, which wasn’t designed to classify diverse skin tones among different groups of people, you would use the Monk Skin Tone Scale, which gives you a more granular understanding,” Monk said. He chose 10 color points to better represent the range of skin tones, opting out of wider range as a larger numbers would begin to create problems with annotation. Extensive fieldwork on skin tone and colorism in the U.S. and Brazil informed the final color selection.

Monk’s collaboration with Google began a few years ago when he was contacted by its Research Responsible AI team, and over time, the work turned to finding an alternative to the Fitzpatrick Scale, the sociologist said. That is when the team learned that Monk had developed a scale of his own.

Google is already using the Monk Skin Tone Scale in its image search. When users search makeup looks, for example, they can refine those searches by skin tone to find more relevant matches. The tech giant also used the scale to update its own Real Tone filters (an initiative it had already launched to deal with the skin-tone problem) on its Pixel cameras so that they can capture a wider, more realistic palette of shades.

“There’ve also been some refinements to search where people have complained that in the past, if you search for certain images, you only get back certain types of people,” Monk said. “One example is if you search for ‘cute babies,’ you might get a completely homogenous set of babies that basically all look white. That’s not a very inclusive experience for people at all.”

Google says the scale will be useful in checking the diversity of results for problems with algorithms. “This has been a longstanding issue, not just at Google, but across the tech industry,” Monk said.

Monk plans to publish research, testing the scale on a diverse group of people. He and the Google team conducted research together to validate the scale in the U.S., which was an extension of his own work.

Developers who do not intentionally design their products to work well across the continuum of skin tones will have products that do not work as well, Monk added. The new scale is open-source so that all companies around the world can make their products more inclusive.

“My hope is that other companies, not just Google, take on the work of auditing and making sure that their products are designed to work equally well across the entire range of skin tones, whether that’s using the Monk Skin Tone Scale or not,” Monk said. “I hope they do that work.”