Grounding Language in Visual Percepts

点击翻译按钮获取中文摘要

The goal of this project is to use perceptual abilities developed in machine vision as foundations for a system that provides dynamic access to useful physical-world knowledge in a form that natural-language understanding and reasoning systems can meaningfully employ in the course of interactive dialogue with human users. To accomplish this goal, mid-level abstractions called "perscripts" are being developed that correspond roughly to the information content of simple phrases and implement an associated class of recognition operations grounded in state-of-the-art machine vision technology. Such abstractions possess sufficient richness and power to bridge the representational gap, while remaining absolutely tied to perceptual processes that operate in interesting real-world environments. The knowledge obtained in the investigation is also of basic interest from a machine vision standpoint since it provides a validated, higher level architecture for visual perception of a sort that is currently lacking in most implemented, real-world systems.The work is being validated in the context of a particular application, a "Household Assistant" whose purpose is to provide people living in a "smart house" with information useful in activities of daily living (e.g., "where did I leave my glasses?" or "did I take my meds?") via a user interface employing natural language dialogue. This application is being developed in conjunction with the University of Rochester Center for Future Health, which has a smart home environment in place, and with the natural language group in the Computer Science Department.This project will provide for the training of graduate students in an area (practical machine vision coupled to natural language interfaces) that has growing economic potential. It will provide basic principles for, and working prototypes of, intelligent computer systems that will stand between people and the physical world. Undergraduates are involved through associated projects. The application in the Center for Future Health addresses the needs of the elderly and the disabled, and has the potential to have a significant economic impact as part of an overall system that unloads caregivers for this population.