CAREER: Simulating Nonlinear Audiovisual Dynamics for Simulated Environments and Interactive Applications

计算机模拟的虚拟动力学越来越被认为是创建沉浸式模拟环境的重要工具，用于理解在真实的实验中研究过于昂贵或危险的物理系统。 然而，大多数模拟技术本质上仍然是沉默的;它们只包含视觉模态，导致对现象的有限感官描述，这与真实的世界形成鲜明对比，在现实世界中，我们不断沉浸在涉及场景照明和听觉线索的丰富视听体验中。 因此，大多数计算机动画，无论它们模拟的效率如何，目前都依赖于手动编辑来添加声音作为事后的想法，许多计算设计工具仍然是面向外观的，仅建模几何或可见运动。 在这个项目中，PI的中心目标是建立有效的模型，以逼真和同步的方式模拟非线性视听动态。 这是一项极具挑战性的任务，因为仿真模型需要在广泛的空间和时间尺度上忠实地解析可见和可听的细节，因为多个运动尺度通常耦合在一起以产生可感知的非线性声音效果，例如音调偏移和模式锁定。 PI将分三个阶段应对这些挑战。 首先，他将创建有效的多尺度模型，自适应地利用潜在的动态结构。 然后，他将使用地面实况数据，实验统计和用户研究系统地评估这些模型。 最后，在他的新模型的基础上，他将开发工具，使交互式参数探索所需的视听效果。这项工作将寻求回答两个基本问题：什么需要被忠实地模拟，以产生视听虚拟动态，以及如何执行模拟在最小的计算成本？ PI旨在为复杂的非线性视听模拟开发有原则的新技术，这将为构建沉浸式模拟环境和交互式应用奠定基础，并连接到其他领域，如计算设计。 在技术层面上，PI的方法将弥合理论工具（包括多尺度分析和动力系统分析）和实际计算方案之间的长期差距。 PI认为，通过将模拟的动态分解为可见和可听的尺度，可以建立适合于各个尺度的有效模拟模型。 该方案可以产生高效率的方法，同时保持模拟保真度。 它也提出了一个新的挑战：在广泛的视听尺度上开发自适应仿真模型。 PI的方法将通过在各个尺度上利用动力系统分析并将所产生的模型连接在一起来解决这一挑战。 如果成功，这项研究将使新的模拟环境应用程序与完全同步的视听效果，并导致新的方式创建和编辑多媒体内容。

Computer simulated virtual dynamics are increasingly recognized as important tools for creating immersive simulated environments for understanding physical systems that are too costly or perilous to investigate in real experiments. Most simulation techniques, however, are still inherently silent; they encompass only the visual modality, resulting in a limited sensory depiction of phenomena which is in sharp contrast to the real world where we are constantly immersed in rich audiovisual experiences involving both scene illumination and auditory cues. Consequently, most computer animations, regardless of how efficiently they are simulated, currently rely on manual editing to add sounds as afterthoughts, and many computational design tools are still appearance-oriented, modeling only geometry or visible motion. In this project the PI's central objective is to build efficient models for simulating nonlinear audiovisual dynamics in a realistic and synchronized way. This is a highly challenging task, as the simulation models need to faithfully resolve visible and audible details across a wide range of spatial and temporal scales because multiple scales of motion are often coupled together to produce perceivable nonlinear sound effects such as pitch shift and mode locking. The PI will address these challenges in three stages. First, he will create efficient multi-scale models which adaptively exploit underlying dynamical structures. He will then systematically evaluate these models using ground-truth data, experimental statistics and user studies. Finally, building upon his new models, he will develop tools to enable interactive parameter exploration of desired audiovisual effects.This work will seek to answer two fundamental questions: what needs to be faithfully simulated in order to produce audiovisual virtual dynamics, and how to perform the simulation at minimal computational cost? The PI seeks to develop principled new techniques for complex nonlinear audiovisual simulations, which will lay the groundwork for building immersive simulated environments and interactive applications, and also for connecting to other areas such as computational design. At a technical level, the PI's approach will bridge the longstanding gap between theoretical tools (including multi-scale analysis and dynamical system analysis) and practical computational schemes. The PI argues that by decomposing the simulated dynamics into visible and audible scales, efficient simulation models adapted to individual scales can be built. This scheme can yield methods with high efficiency while retaining simulation fidelity. It also poses a new challenge: to develop adaptive simulation models across a wide range of audiovisual scales. The PI's approach will address this challenge by exploiting dynamical system analysis at individual scales and linking the resulting models together. If successful, this research will enable new simulated environment applications with fully synchronized audiovisual effects, and lead to new ways of creating and editing multimedia content.