CHS: Small: Teachable Object Recognizers for the Blind

识别物体，从食品包装到衣服，是我们主要使用视觉执行的日常任务。 盲人在无法获得视力帮助的情况下，尝试使用感官、策略和特设组织系统的组合来完成这些任务。 近年来，盲人社区的成员已经成为使用图像识别来识别物体的移动的应用程序的早期采用者。 然而，这样的解决方案目前对于许多感兴趣的对象具有有限的使用，因为图像识别器不能提供足够的粒度来区分所有用户之间的所有可能的感兴趣的对象。 他们还倾向于接受视力正常的人拍摄的图像的训练，这些图像具有与盲人用户拍摄的照片不同的背景杂乱，比例，视点，遮挡和图像质量。 这项研究的目标是使盲人用户能够通过一种称为可教性的新方法来定制识别任务，以适应他们感兴趣的对象，拍照策略和环境，这种方法有望允许非专家的最终用户为这些应用程序中的机器学习模型提供训练示例。 具体来说，将设计、部署和公开发布一个可教对象识别器（TOR）应用程序，盲人用户可以通过智能手机的摄像头提供标签沿着一些对象的例子来进行训练。 如果成功，项目成果将产生广泛影响，通过赋予残疾人定义智能辅助技术功能的能力，改变智能辅助技术的性质，特别是在小型识别任务中。该项目通过可教性解决无障碍方面的数据稀缺问题，最终用户教授预先训练的模型，这些模型使用更多可用但相关性较低的数据，较少但更相关的数据，以满足他们的需求。 这项工作将在盲人物体识别的背景下研究可教性的概念，并将调查无障碍研究是否可以利用盲人用户有限的数据利用计算机视觉的进步。 需要探讨的问题包括：如何向用户最好地解释这种智能系统，以获得更高质量的训练数据？衡量其功效的最佳方法是什么？哪些设计参数、传感模式、交互和算法对它们的成功影响最大？该项目将包括各种研究方法：调查，参与式设计会议，原型可用性测试，基于实验室的用户研究，以及盲人用户的纵向现实世界评估。 利用可教对象识别的工作原型移动的应用程序，它将调查学习培训的无障碍交互，并检查这种可教辅助技术的鲁棒性和可扩展性可以提高的基本机制。该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Identifying objects, from packages of food to items of clothing, is an everyday task that we perform predominantly using our sense of sight. Blind persons, for whom sighted help is not available, attempt such tasks using a combination of senses, strategies, and ad hoc organizing systems. In recent years, members of the blind community have been early adopters of mobile apps that use image recognition for identifying objects. However, such solutions currently have limited use for many objects of interest, because image recognizers cannot provide enough granularity to distinguish among all possible objects of interest across all users. They also tend to be trained on images taken by sighted people with different background clutter, scale, viewpoints, occlusion, and image quality than in photos taken by blind users. The goal of this research is to empower blind users to customize the recognition task to their objects of interest, photo-taking strategies, and environment, through a new approach called teachability which holds the promise of allowing end users that are non-experts to provide the training examples for the machine learning models in these applications. Specifically, a teachable object recognizer (TOR) app will be designed, deployed and publicly released that blind users can train by providing labels along with a few examples of their objects through a smartphone's camera. If successful, project outcomes will have broad impact by changing the nature of smart assistive technologies by empowering people with disabilities to define the functionality of such technologies, especially for small recognition tasks.This project addresses the data scarcity issue in accessibility through teachability, where end users teach models pretrained on more available yet less relevant data, with fewer but more pertinent data specific to their needs. The work will examine the concept of teachability in the context of object recognition for the blind, and will investigate whether accessibility research can leverage advances in computer vision with limited data from blind users. Questions to be explored will include: How to best explain such intelligent systems to users for higher quality training data? What is the best way to measure their efficacy? What design parameters, sensing modalities, interactions, and algorithms are most influential on their success? The project will include a variety of research methods: surveys, participatory design sessions, prototype usability testing, lab-based user studies, and longitudinal real-world evaluation with blind users. Using a working prototype mobile application on teachable object recognition, it will investigate accessible interactions on learning-to-train and examine the underlying mechanisms by which robustness and scalability of such teachable assistive technologies can be improved.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.