Improvising with the dead

这项拟议研究的最终目标很容易说明。我想开发技术来支持目前不可能的即兴音乐体验，与一位或多位已经去世的音乐家一起。这种体验应该尽可能接近实际的音乐播放。虽然这一目标很容易说明，但它的实现是具有挑战性的，需要在几个学科的不同研究主题上取得重要进展。 即兴创作存在于所有音乐文化中，即使是在他们不熟悉的文化中，也能被大多数音乐听众识别和欣赏。支配即兴创作的过程还不是很清楚，而且还取决于所研究的特定音乐文化。它们还与演奏乐器的身体动作内在地联系在一起。例如，要理解和欣赏B.B·金的独奏曲，光看演奏的音符是不够的，还需要考虑他的身体和吉他之间的内在联系。我认为，我们正处于这样一个时代，与一位已故音乐家即兴创作的经历正在从科幻小说转向可能的领域，因为最近有几项关键技术发展可以用于这一目的。想象一下，能够分析一位传奇爵士音乐家的录音，例如令人难以置信的萨克斯管演奏家查理·帕克，以便创建他如何演奏和即兴演奏的模型。该模型可以用来创建一个计算机控制的虚拟查理·帕克，爵士乐学生可以与之玩耍并从中学习。为了尽可能准确地接近演奏体验，在演奏实际的声学乐器时将使用增强现实或虚拟现实(AR/VR)设备。 AR/VR卷土重来，涉及到许多关键的行业参与者。在音乐表达社区的新界面中，有为音乐构建有效的网络物理系统的丰富经验。音乐信息检索(MIR)技术通过分析数百万首歌曲来提取内容信息，从而自动推荐音乐。MIR技术还可以用于实时提取有关正在播放的内容的信息。我的小组一直活跃在这些研究领域，并将以这一新的重点继续这样做。从事这些项目的学生将获得音频信号处理、机器学习、增强和虚拟现实、人机交互(HCI)、编程语言和人工智能(AI)方面的经验。随着虚拟词汇和物理词汇的融合，新的互动方式将被开发出来。拟议的工作还可以导致人工智能和用于构建复杂、实时的人工智能系统的工具的进步，并塑造人类与此类系统未来的互动方式。最后，这项提案中设想的进步有可能从根本上改变我们未来创建、分发和感知音乐(以及一般的多媒体)内容的方式。

The end goal of the proposed research is rather simple to state. I would like to develop technologies to support the currently impossible experience of improvising music together with one or more musicians that are dead. This experience should be as close to actual music playing as possible. Although this goal is simple to state it is challenging to achieve and requires important advances in different research topics from several disciplines. Improvisation is present in all music cultures and can be identified and appreciated by most music listeners even in cultures they are not familiar with. The processes that govern improvisation are not well understood and also depend on the specific music culture studied. They are also intrinsically tied to the physical actions of playing instruments. For example to understand and appreciate a solo by B.B King it is not sufficient to look at what notes are played but one also needs to consider the visceral connection between his body and the guitar. I believe that we are at a time when the experience of improvising with a dead musician is moving from science fiction to the realm of possibility as there are several recent key technological developments that can be leveraged for this purpose. Imagine being able to analyze the audio recordings of a legendary jazz musician, for example the incredible saxophone player Charlie Parker, in order to create a model of how he played and improvised. The model could be used to create a computer controlled virtual Charlie Parker that a jazz student could play with and learn from. In order to approximate the playing experience as accurately as possible augmented or virtual reality (AR/VR) devices would be used while playing an actual acoustic instrument. There is resurgence of AR/VR with many key industrial players involved. In the new interfaces for musical expression community there is extensive experience building effective cyber-physical systems for music. Music information retrieval (MIR) techniques analyze millions of songs to extract content information to automatically recommend music. MIR techniques can also be used to extract information about what is being played in real time. My group has been active in these research areas and will continue to do so with this new focus. Students working on these project will obtain experience in audio signal processing, machine learning, augmented and virtual reality, human- computer interaction (HCI), programming languages and artificial intelligence (AI). As the virtual and physical words blend, new ways of interaction will be developed. The proposed work can also lead to advances in AI and the tools used to build complicated, real-time, AI systems and shape how humans and such systems interact in the future. Finally, the advances envisioned in this proposal have the potential to radically transform how we create, distribute and perceive music (and multimedia in general) content in the future.