Musical Multimodal Interaction
Musical Multimodal Interaction
复制标题
音乐多模态交互
DOI：
10.1145/3382507.3419444
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Tanaka A
中科院分区：
文献类型：
--
作者：
Tanaka A
音乐表演可以被认为是多模态的-与乐器的物理交互产生声音输出，通常是当表演者在视觉上阅读乐谱时。数字乐器（Digital Musical Instrument，缩写为NIM）设计融合了HCI和乐器实践的原则。视听表演和其他形式的多媒体可能受益于多模态思维。这个主题回顾了二十年的互动音乐实践，促进了多模态互动研究领域的发展。BioMuse是一种早期的数字乐器系统，使用EMG肌肉传感，通过第二种传感模式进行扩展，使努力和位置成为两种互补的模式[1]。Haptic Wave应用跨模态信息显示的原理来创建触觉音频编辑器，使视觉受损的音频制作者能够“感受”他们在图形用户界面中看不到的音频波形[2]。VJ文化将音乐DJ的理念延伸到创造视听文化体验。AVUI是一套创造性的编码工具，它可以实现性能UI和创造性视觉输出的融合[3]。岩石乐团是与视觉艺术家Uta Kogelsberger的持续合作，通过物理和虚拟形式表现出来-随着时间的推移允许多模态[4]。无论是画廊中的实体展览还是YouTube 360上的音频反应式3D动画，艺术品的多种表达模式都支持其原始的概念基础。这四个项目将多模态互动置于艺术研究的核心。
Musical performance can be thought of in multimodal terms - physical interaction with musical instruments produces sound output, often while the performer is visually reading a score. Digital Musical Instrument (DMI) design merges tenets of HCI and musical instrument practice. Audiovisual performance and other forms of multimedia might benefit from multimodal thinking. This keynote revisits two decades of interactive music practice that has paralleled the development of the field of multimodal interaction research. The BioMuse was an early digital musical instrument system using EMG muscle sensing that was extended by a second mode of sensing, allowing effort and position to be two complementary modalities [1]. The Haptic Wave applied principles of cross-modal information display to create a haptic audio editor enabling visually impaired audio producers to 'feel' audio waveforms they could not see in a graphical user interface [2]. VJ culture extends the idea of music DJs to create audiovisual cultural experiences. AVUIs were a set of creative coding tools that enabled the convergence of performance UI and creative visual output [3]. The Orchestra of Rocks is a continuing collaboration with visual artist Uta Kogelsberger that has manifested itself through physical and virtual forms - allowing multimodality over time [4]. Be it a physical exhibition in a gallery or audio reactive 3D animation on YouTube 360, the multiple modes in which an artwork is articulated support its original conceptual foundations. These four projects situate multimodal interaction at the heart of artistic research.