CHS: Small: Teachable Object Recognizers for the Blind

点击翻译按钮获取中文摘要

Identifying objects, from packages of food to items of clothing, is an everyday task that we perform predominantly using our sense of sight. Blind persons, for whom sighted help is not available, attempt such tasks using a combination of senses, strategies, and ad hoc organizing systems. In recent years, members of the blind community have been early adopters of mobile apps that use image recognition for identifying objects. However, such solutions currently have limited use for many objects of interest, because image recognizers cannot provide enough granularity to distinguish among all possible objects of interest across all users. They also tend to be trained on images taken by sighted people with different background clutter, scale, viewpoints, occlusion, and image quality than in photos taken by blind users. The goal of this research is to empower blind users to customize the recognition task to their objects of interest, photo-taking strategies, and environment, through a new approach called teachability which holds the promise of allowing end users that are non-experts to provide the training examples for the machine learning models in these applications. Specifically, a teachable object recognizer (TOR) app will be designed, deployed and publicly released that blind users can train by providing labels along with a few examples of their objects through a smartphone's camera. If successful, project outcomes will have broad impact by changing the nature of smart assistive technologies by empowering people with disabilities to define the functionality of such technologies, especially for small recognition tasks.This project addresses the data scarcity issue in accessibility through teachability, where end users teach models pretrained on more available yet less relevant data, with fewer but more pertinent data specific to their needs. The work will examine the concept of teachability in the context of object recognition for the blind, and will investigate whether accessibility research can leverage advances in computer vision with limited data from blind users. Questions to be explored will include: How to best explain such intelligent systems to users for higher quality training data? What is the best way to measure their efficacy? What design parameters, sensing modalities, interactions, and algorithms are most influential on their success? The project will include a variety of research methods: surveys, participatory design sessions, prototype usability testing, lab-based user studies, and longitudinal real-world evaluation with blind users. Using a working prototype mobile application on teachable object recognition, it will investigate accessible interactions on learning-to-train and examine the underlying mechanisms by which robustness and scalability of such teachable assistive technologies can be improved.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.