CAREER: Simulating Nonlinear Audiovisual Dynamics for Simulated Environments and Interactive Applications

职业：模拟模拟环境和交互式应用的非线性视听动力学

• 批准号: 1453101
• 负责人: Changxi Zheng
• 金额: $ 50.18万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-02-01 至 2021-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1453101&HistoricalAwards=false
• 关键词: CAREER; Simulating; Nonlinear; Audiovisual; Dynamics

Computer simulated virtual dynamics are increasingly recognized as important tools for creating immersive simulated environments for understanding physical systems that are too costly or perilous to investigate in real experiments. Most simulation techniques, however, are still inherently silent; they encompass only the visual modality, resulting in a limited sensory depiction of phenomena which is in sharp contrast to the real world where we are constantly immersed in rich audiovisual experiences involving both scene illumination and auditory cues. Consequently, most computer animations, regardless of how efficiently they are simulated, currently rely on manual editing to add sounds as afterthoughts, and many computational design tools are still appearance-oriented, modeling only geometry or visible motion. In this project the PI's central objective is to build efficient models for simulating nonlinear audiovisual dynamics in a realistic and synchronized way. This is a highly challenging task, as the simulation models need to faithfully resolve visible and audible details across a wide range of spatial and temporal scales because multiple scales of motion are often coupled together to produce perceivable nonlinear sound effects such as pitch shift and mode locking. The PI will address these challenges in three stages. First, he will create efficient multi-scale models which adaptively exploit underlying dynamical structures. He will then systematically evaluate these models using ground-truth data, experimental statistics and user studies. Finally, building upon his new models, he will develop tools to enable interactive parameter exploration of desired audiovisual effects.This work will seek to answer two fundamental questions: what needs to be faithfully simulated in order to produce audiovisual virtual dynamics, and how to perform the simulation at minimal computational cost? The PI seeks to develop principled new techniques for complex nonlinear audiovisual simulations, which will lay the groundwork for building immersive simulated environments and interactive applications, and also for connecting to other areas such as computational design. At a technical level, the PI's approach will bridge the longstanding gap between theoretical tools (including multi-scale analysis and dynamical system analysis) and practical computational schemes. The PI argues that by decomposing the simulated dynamics into visible and audible scales, efficient simulation models adapted to individual scales can be built. This scheme can yield methods with high efficiency while retaining simulation fidelity. It also poses a new challenge: to develop adaptive simulation models across a wide range of audiovisual scales. The PI's approach will address this challenge by exploiting dynamical system analysis at individual scales and linking the resulting models together. If successful, this research will enable new simulated environment applications with fully synchronized audiovisual effects, and lead to new ways of creating and editing multimedia content.

计算机模拟的虚拟动力学越来越被认为是创建沉浸式模拟环境的重要工具，用于理解在真实实验中过于昂贵或危险的物理系统。然而，大多数模拟技术本质上仍然是沉默的；它们只包含视觉形态，导致对现象的有限感官描述与现实世界形成鲜明对比，在现实世界中，我们不断沉浸在丰富的视听体验中，包括场景照明和听觉线索。因此，大多数计算机动画，无论它们的模拟效率如何，目前都依赖于手动编辑来添加声音，并且许多计算设计工具仍然是面向外观的，只建模几何或可见运动。在这个项目中，PI的中心目标是以现实和同步的方式建立模拟非线性视听动态的有效模型。这是一项极具挑战性的任务，因为模拟模型需要在大范围的空间和时间尺度上忠实地解析可见和可听细节，因为多个运动尺度经常耦合在一起，以产生可感知的非线性声音效果，如音调移位和模式锁定。该计划将分三个阶段应对这些挑战。首先，他将创建有效的多尺度模型，自适应地利用潜在的动态结构。然后，他将使用真实数据、实验统计和用户研究系统地评估这些模型。最后，建立在他的新模型，他将开发工具，使交互式参数探索所需的视听效果。这项工作将寻求回答两个基本问题：为了产生视听虚拟动力学，需要忠实地模拟什么，以及如何以最小的计算成本执行模拟？PI寻求为复杂的非线性视听模拟开发原则性的新技术，这将为构建沉浸式模拟环境和交互式应用奠定基础，也为连接其他领域（如计算设计）奠定基础。在技术层面上，PI的方法将弥合理论工具（包括多尺度分析和动力系统分析）与实际计算方案之间长期存在的差距。PI认为，通过将模拟的动态分解为可见和可听的尺度，可以建立适合单个尺度的有效模拟模型。该方案在保证仿真逼真度的前提下，产生了高效率的方法。这也提出了一个新的挑战：在广泛的视听尺度上开发自适应模拟模型。PI的方法将通过在单个尺度上利用动力系统分析并将结果模型连接在一起来解决这一挑战。如果成功，这项研究将使新的模拟环境应用具有完全同步的视听效果，并导致创建和编辑多媒体内容的新方法。