Collaborative Research: CNS Core: Small: SmartSight: an AI-Based Computing Platform to Assist Blind and Visually Impaired People

该项目将设计SmartSight，这是一个将设备到云连续体的计算能力与人工智能（AI）方法相结合的平台，为盲人和视障人士（VIP）提供环境感知，从而提高他们的生活质量和社会包容性。SmartSight基于智能眼镜运行，通过跨从本地层（设备和边缘）到远程层（雾和云平台）的连续层高效部署机器学习和近似计算方法，为其用户提供实时感知。SmartSight使用联邦学习来利用用户的集体观察并提高其感知准确性。该平台在与远程层的通信中具有强大的抗故障能力，并且能够仅使用本地层提供其基本功能。为此，SmartSight动态调整近似计算旋钮，并将关键任务分配给资源，前提是这些任务能够及时产生结果。将开发SmartSight原型，并在盲态和VIP中进行测试，以评估其可用性、稳健性、便利性和离散性。SmartSight实现“AI+系统为社会公益服务”。该项目的成果不仅将影响科学发现的前沿，还将通过改善盲人和VIP的生活质量和社会包容性来影响社会。其他患有某种形式的认知能力下降的公民（例如，老年痴呆症）将受益于SmartSight技术，并能够作为社会的独立成员发挥作用。解决这些障碍的步骤不仅将提高这些人的生活质量，而且还将为整个社会节省大量费用。从全球范围来看，潜在的收益甚至更大。将为SmartSight平台开发的智能资源分配和近似计算方法可以适用于其他软实时计算系统，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准。

This project will design SmartSight, a platform that combines the computing ability of the device-to-cloud continuum with Artificial Intelligence (AI) methods to enable ambient perception for blind and visually impaired people (VIP), thereby improving their quality of life and social inclusion. SmartSight operates based on smart glasses and provides real-time perception for its user by efficiently deploying machine learning and approximate computing methods across a continuum of tiers—from local tiers (device and edge) to remote tiers (fog and cloud platforms). SmartSight uses federated learning to take advantage of collective observations of users and improve its perception accuracy. The platform is robust against failure in its communication with remote tiers and is able to provide its basic functionality using only local tiers. For that purpose, SmartSight dynamically adjusts approximate computing knobs and allocates critical tasks to resources, only if they can produce timely results. A prototype of SmartSight will be developed and tested with blind and VIP to assess its usability, robustness, convenience, and discreteness. SmartSight enables “AI+Systems for social good.” The project results will impact not only the forefront of scientific discovery, but also society by improving the quality of life and social inclusion of the blind and VIP. Other citizens who suffer from some form of cognitive decline (e.g., Alzheimer's disease) will benefit from SmartSight technology and be able to function as independent members of society. The steps towards addressing these hurdles not only will improve the quality of life for these people but will also bring about a remarkable cost-saving to society as a whole. The potential gain is even greater when considered on a global scale. The intelligent resource allocation and approximate computing methods that will be developed for the SmartSight platform can be adapted to other soft real-time computing systems, and the proposed efficient and scalable approaches for automated parameter tuning and ubiquitous model execution can be employed in other tunable software systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.