Grounding Language in Visual Percepts

该项目的目标是使用机器视觉中开发的感知能力作为系统的基础，该系统以自然语言理解和推理系统可以在与人类用户的交互对话过程中有意义地使用的形式提供对有用的物理世界知识的动态访问。 为了实现这一目标，正在开发称为“perscripts”的中级抽象，其大致对应于简单短语的信息内容，并实现基于最先进的机器视觉技术的相关识别操作类。 这种抽象具有足够的丰富性和力量来弥合表征的差距，同时保持绝对依赖于在有趣的现实世界环境中操作的感知过程。 从机器视觉的角度来看，在调查中获得的知识也是基本的兴趣，因为它提供了一个有效的，更高层次的视觉感知体系结构，这是目前大多数实现的，现实世界的系统所缺乏的。其目的是向生活在“智能住宅”中的人们提供在日常生活活动中有用的信息的“家庭助理”（例如，“我把眼镜放哪儿了？或者“我吃药了吗？“）通过使用自然语言对话的用户界面。 该应用程序是与罗切斯特大学未来健康中心（该中心拥有智能家居环境）以及计算机科学系的自然语言组合作开发的。该项目将为研究生提供一个具有增长经济潜力的领域（实用机器视觉与自然语言接口相结合）的培训。 它将为介于人类和物理世界之间的智能计算机系统提供基本原理和工作原型。 本科生通过相关项目参与。 未来健康中心的应用程序解决了老年人和残疾人的需求，并有可能产生重大的经济影响，作为整个系统的一部分，为这一人群卸载护理人员。

The goal of this project is to use perceptual abilities developed in machine vision as foundations for a system that provides dynamic access to useful physical-world knowledge in a form that natural-language understanding and reasoning systems can meaningfully employ in the course of interactive dialogue with human users. To accomplish this goal, mid-level abstractions called "perscripts" are being developed that correspond roughly to the information content of simple phrases and implement an associated class of recognition operations grounded in state-of-the-art machine vision technology. Such abstractions possess sufficient richness and power to bridge the representational gap, while remaining absolutely tied to perceptual processes that operate in interesting real-world environments. The knowledge obtained in the investigation is also of basic interest from a machine vision standpoint since it provides a validated, higher level architecture for visual perception of a sort that is currently lacking in most implemented, real-world systems.The work is being validated in the context of a particular application, a "Household Assistant" whose purpose is to provide people living in a "smart house" with information useful in activities of daily living (e.g., "where did I leave my glasses?" or "did I take my meds?") via a user interface employing natural language dialogue. This application is being developed in conjunction with the University of Rochester Center for Future Health, which has a smart home environment in place, and with the natural language group in the Computer Science Department.This project will provide for the training of graduate students in an area (practical machine vision coupled to natural language interfaces) that has growing economic potential. It will provide basic principles for, and working prototypes of, intelligent computer systems that will stand between people and the physical world. Undergraduates are involved through associated projects. The application in the Center for Future Health addresses the needs of the elderly and the disabled, and has the potential to have a significant economic impact as part of an overall system that unloads caregivers for this population.