Controlling Generative Musical Systems: Getting the Right Data & Using the Right Instrument

自2020年以来，可控生成模型的进展一直是惊人的，包括可以生成文本的大型语言模型，以及最近的文本到图像模型，其中用户提供文本，系统将产生像素值以创建新的相应图像。对这种强大模型的研究兴趣正在转向音乐和音频。因此，有一个紧迫的机会，以加快研究基于ML的生成模型和自适应工具的音乐，这一建议将使在这一方向的重要进展。该提案集中在一个设备上，该设备是两个主要研究方向的基础：（1）研究方向：音乐的可控生成模型。动机：（a）大型语言模型和（B）Dalle-2，Stable Diffusion，Craiyon的惊人成功显示了强大的生成模型在文本和视觉领域的影响。音乐领域的成功也将产生巨大的影响。障碍：可控的生成音乐模型目前受到缺乏良好注释数据的限制。与提案的关系：拟议的设备将允许精确的高质量数据收集所需的训练有效和可控的音乐生成模型。（2）研究方向：支持创造力的自适应和交互式音乐工具。动机：首先，上述生成模型的成功显示了创造力支持工具的影响。其次，最近自我监督预训练与生成模型相结合的成功意味着机器学习系统的人在回路框架已经成熟，可以探索，自适应乐器是探索这个框架的理想背景。障碍物：构建一种有效的交互式乐器需要高质量的硬件，创新的机器学习算法将在其上运行。构建一个专家想要使用的人在回路系统，首先需要为人类提供一个可以强大控制的工具。与提案的关系：拟议的设备将提供高质量的乐器，作为建立自适应音乐工具的基础。建议设备：这两个项目的主要设备是一台能够准确记录钢琴家动作的钢琴，它也可以通过执行一组给定的动作来“演奏自己”。它可以执行这些操作，无论它们是由人记录的，还是由模型生成的。这将使高质量的数据收集，数据注释，并在其上的适应性乐器的发展进行验证的基础。一个有10名HQP参与的试点项目表明了这一建议的可行性、必要性和潜在的重大影响。

Since roughly 2020, progress in controllable generative models has been phenomenal, and has included large language models that can generate text, and recently text-to-image models, where a user provides text and the system will produce pixel values to create a new corresponding image. Research interest on such powerful models is turning towards music and audio. There is thus an urgent opportunity to accelerate research in ML-based generative models and adaptive tools for music, and this proposal will enable essential progress in this direction. This proposal centres on one piece of equipment that is foundational for two primary research directions: (1) Research direction: Controllable generative models for music. Motivation: The phenomenal success of (a) large language models, and (b) Dalle-2, Stable Diffusion, Craiyon shows the impact of powerful generative models in both text and visual domains. Success in the music domain will also have enormous impact. Obstacle: Controllable generative music models are currently limited by lack of well-annotated data. Relationship to Proposal: The proposed equipment will allow exactly the high-quality data collection needed for training effective and controllable generative models for music. (2)Research direction: Adaptive and interactive musical tools for supporting creativity. Motivation: First, the success of the generative models described above shows the impact of creativity-support tools. Second, the recent success of self-supervised pre-training combined with that of generative models means that the framework of human-in-the-loop for machine learning systems is ripe for exploration, and adaptive musical instruments are the ideal context for exploring this framework. Obstacle: Building an interactive musical instrument that will be effective requires a high-quality hardware on which the innovative machine-learning algorithms will be running. Building a human-in-the-loop system that an expert human will want to use requires providing the human with a tool that can be powerfully controlled in the first place. Relationship to Proposal: The proposed equipment will provide exactly the high-quality instrument that is required as a foundation for building an adaptive musical tool. Proposed Equipment: The single piece of equipment that will be the mainstay for both of these projects is the Disklavier PRO, a piano that can both accurately record the pianist's actions, and also can "play itself" by performing a given set of actions. It can perform these actions whether they were were recorded by a person, or whether they were generated by a model. This will enable high-quality data collection, data annotation, and the foundation upon which to iterate the development of an adaptive musical instrument. A pilot project with 10 HQP has indicated the feasibility, need, and potential major impact of this proposal.