CHS: Small: Teachable Object Recognizers for the Blind

识别物品，从食物包装到衣物，是我们主要利用视觉完成的一项日常任务。盲人无法获得视力帮助，他们尝试使用感官、策略和特别组织系统的组合来完成这类任务。近年来，盲人社区的成员已经成为使用图像识别来识别物体的移动应用程序的早期采用者。然而，这种解决方案目前对许多感兴趣对象的使用有限，因为图像识别器不能提供足够的粒度来在所有用户之间区分所有可能的感兴趣对象。他们也倾向于在有视力的人拍摄的图像上接受训练，这些图像的背景杂乱、比例、视点、遮挡和图像质量与盲人用户拍摄的照片不同。这项研究的目标是使盲人用户能够根据他们感兴趣的对象、拍照策略和环境来定制识别任务，通过一种名为可教学性的新方法，该方法有望允许非专家用户为这些应用中的机器学习模型提供训练样本。具体地说，将设计、部署和公开发布一款可教对象识别器(TOR)应用程序，盲人用户可以通过智能手机的摄像头提供标签和几个对象示例来进行培训。如果成功，项目成果将产生广泛的影响，改变智能辅助技术的性质，使残疾人能够定义此类技术的功能，特别是对于小型识别任务。该项目通过可教学性解决数据稀缺问题，即最终用户教授预先培训的模型，使用更多但更相关的数据来满足他们的需求。这项工作将在盲人物体识别的背景下研究可教性的概念，并将调查可访问性研究是否可以利用来自盲人用户的有限数据来利用计算机视觉的进步。将探讨的问题包括：如何最好地向用户解释这种智能系统，以获得更高质量的培训数据？衡量其有效性的最佳方法是什么？哪些设计参数、感知方式、交互作用和算法对它们的成功影响最大？该项目将包括各种研究方法：调查、参与式设计会议、原型可用性测试、基于实验室的用户研究，以及针对盲人用户的纵向真实世界评估。使用一个关于可教对象识别的工作原型移动应用程序，它将调查关于学习到培训的可访问交互，并检查可以改善此类可教辅助技术的健壮性和可扩展性的潜在机制。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Identifying objects, from packages of food to items of clothing, is an everyday task that we perform predominantly using our sense of sight. Blind persons, for whom sighted help is not available, attempt such tasks using a combination of senses, strategies, and ad hoc organizing systems. In recent years, members of the blind community have been early adopters of mobile apps that use image recognition for identifying objects. However, such solutions currently have limited use for many objects of interest, because image recognizers cannot provide enough granularity to distinguish among all possible objects of interest across all users. They also tend to be trained on images taken by sighted people with different background clutter, scale, viewpoints, occlusion, and image quality than in photos taken by blind users. The goal of this research is to empower blind users to customize the recognition task to their objects of interest, photo-taking strategies, and environment, through a new approach called teachability which holds the promise of allowing end users that are non-experts to provide the training examples for the machine learning models in these applications. Specifically, a teachable object recognizer (TOR) app will be designed, deployed and publicly released that blind users can train by providing labels along with a few examples of their objects through a smartphone's camera. If successful, project outcomes will have broad impact by changing the nature of smart assistive technologies by empowering people with disabilities to define the functionality of such technologies, especially for small recognition tasks.This project addresses the data scarcity issue in accessibility through teachability, where end users teach models pretrained on more available yet less relevant data, with fewer but more pertinent data specific to their needs. The work will examine the concept of teachability in the context of object recognition for the blind, and will investigate whether accessibility research can leverage advances in computer vision with limited data from blind users. Questions to be explored will include: How to best explain such intelligent systems to users for higher quality training data? What is the best way to measure their efficacy? What design parameters, sensing modalities, interactions, and algorithms are most influential on their success? The project will include a variety of research methods: surveys, participatory design sessions, prototype usability testing, lab-based user studies, and longitudinal real-world evaluation with blind users. Using a working prototype mobile application on teachable object recognition, it will investigate accessible interactions on learning-to-train and examine the underlying mechanisms by which robustness and scalability of such teachable assistive technologies can be improved.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.