Hoping to help people who are blind, professors from UT are collaborating with Microsoft to create a dataset that can be used by the public to expand automated image captioning.

The goal of the Microsoft Ability Initiative is to use artificial intelligence to make life more accessible to people who are blind or have low vision by having machines verbally describe pictures for them, principal investigator Danna Gurari said.

“It’s very important that I do something that feels like it has social impact,” Gurari said. “While I’m a computer scientist by training, I very much care about the connection to the real world.”

The first part of the initiative aims to develop a public dataset to create image captioning algorithms, said Microsoft principal researcher Meredith Morris. 

“We’ll be following that up with additional interviews and studies with people who are blind to understand more about how we can pillar the data and use algorithms to end user’s specific needs,” Morris said.

Now, researchers are asking paid volunteers to describe images taken and submitted by people who are blind through the crowdsourcing marketplace Amazon Mechanical Turk. They hope this data will help teach computers to describe similar images.

The goal is to have data collected on 39,000 images, Gurari said. By early May, Gurari hopes to have finished the public dataset, which will then be used by Microsoft to create a “novel algorithm to create captions.”

The initiative is interdisciplinary across technical and social science fields. Working alongside Gurari as a co-principal investigator, social scientist Kenneth R. Fleischmann said he focuses on human values within the technology.

“In the same image, different people in different situations might want different captions,” Fleischmann said. “Trying to make sure that the caption is not just a generic caption that works fairly well for everyone, but ideally to be able to customize it to some degree to align with the values of the user (is the goal).”

Fleischmann is also involved in a project called Good Systems, which is an initiative that aims to ensure artificial intelligence benefits society. He said the Microsoft Ability Initiative is similar. 

“The whole purpose of Good Systems is to ensure that AI is making the world a better place and is helping people,” Fleischmann said. “That’s exactly what we’re doing in this project. We’re using AI to help people.”