CAREER: Large-scale Appearance Modeling

我们周围世界的视觉外观是组成场景的不同表面和材料属性之间复杂的光相互作用的结果。尽管在数据驱动的外观建模方面取得了惊人的进步，但创建大型环境的精确模型仍然是一个悬而未决的问题。目前大多数外观建模方法依赖于主动照明来探测场景外观的不同切片，这使得它们无法在有限或无法控制入射环境照明的环境中使用。此外，为了便于校准，许多外观建模技术从固定的有利位置估计场景的外观，排除了太大而无法在单个视图中容纳足够细节的场景。在这项研究中，PI将研究两种新颖的外观建模范例，这些范例是专门为不受控制的环境照明下的大规模环境设计的：运动外观和相似性外观。前者利用来自不同观点的观察结果之间的关系来推断全反射行为，而后者则试图从已有的外观实例库中确定最佳匹配，以可能是一组约束不足的观察结果。为了支持这两种范式，将开发一种新的外观模型，该模型建立在我们对场景外观的直觉之上。这项工作将集中于两种常见的输入类型：社区照片收集和有针对性的视频序列。更广泛的影响：这项研究将为大规模环境的现场外观建模的实用技术铺平道路，同时通过回答我们是否可以通过运动和/或利用相似性来建模外观的基本问题，刺激计算机视觉和计算机图形学中数据驱动外观建模的新研究。该项目不仅将对计算机科学产生深远的影响，而且将对城市规划、文化遗产、娱乐等各个领域产生深远的影响。模拟现有环境的能力将有利于各种安全和安全培训计划（例如，现有建筑物和现场的虚拟消防演习模拟可以帮助培训和准备消防员和急救人员）。新兴的虚拟现实治疗领域也将受益于这项研究，因为它可以更容易地创建大规模环境的数字模型（例如，中风患者可以在他们日常生活中遇到的环境的虚拟复制品中练习运动康复技能）。而自闭症儿童可以在他们日常生活中遇到的地方（如教室）的虚拟复制品中进行训练，以改善他们的社会互动。

The visual appearance of the world around us is the result of complex light interactions between different surfaces and material properties that comprise a scene. Despite staggering advances in data-driven appearance modeling, the creation of accurate models of large environments remains an open problem. The reliance of most current appearance modeling methods on active lighting to probe different slices of a scene's appearance precludes their use in environments where there is limited or no control over the incident ambient lighting. Furthermore, to facilitate calibration, many appearance modeling techniques estimate the appearance of a scene from a fixed vantage point, excluding scenes too large to fit in a single view with sufficient detail. In this research, the PI will investigate two novel appearance modeling paradigms designed expressly for large-scale environments under uncontrolled ambient lighting: appearance-from-motion and appearance-by-similarity. The former exploits relations between observations from different viewpoints to infer the full reflectance behavior, while the latter seeks to identify the best match from a library of pre-existing appearance instances to a possibly under-constrained set of observations. To support these two paradigms, a novel appearance model will be developed that builds upon our intuitions regarding scene appearance. The work will focus on two common types of input: community photo-collections and targeted video sequences.Broader Impacts: This research will pave the way towards practical techniques for in-situ appearance modeling of large-scale environments, while stimulating new research in computer vision and in data-driven appearance modeling in computer graphics by answering fundamental questions as to whether we can model appearance from motion and/or by exploiting similarity. The project will have far-reaching impact not only on computer science but also on diverse fields ranging from metropolitan planning to cultural heritage to entertainment. The ability to model existing environments will be beneficial to various security and safety training programs (for example, virtual fire drill simulations of existing buildings and sites could help train and prepare firefighters and first responders). The emerging field of virtual reality therapy will also benefit from this research, by making it easier to create digital models of large-scale environments (so that, for example, patients who have suffered a stroke can practice motor rehabilitation skills in virtual reproductions of environments they encounter in their daily lives, while autistic children can train to improve their social interactions in virtual reproductions of places such as classrooms which they encounter in their daily lives).