SGER: Decoding the Human Conducting Gesture

音乐指挥是一种复杂的手势交流形式。尽管经历了几个世纪的实践，但甚至对于信息传递的内容也没有达成共识，更不用说信息是如何编码和解码的。该项目在通过仪器和测量进行指挥的性质以及在感知、检测和解释人类手势的技术方面取得了同步进展。该项目使用可穿戴的无线传感器记录手势，并将手势数据与视频和音频同步。该项目将与音乐家和指挥家密切合作，开发关于指挥手势的新假设，开发新的传感器来捕获相关信息，然后使用机器学习来创建指挥手势识别系统。结果是对导体传达的信息进行了表征，并改进了一代可穿戴传感器和感知人体运动的技术。这项工作的更广泛影响包括扩大实验科学在艺术中的作用，特别是音乐。通过与专业交响乐团和学生交响乐团的合作，研究的方法和好处将清楚地传达给非科学家。这项工作也将通过更好地了解指挥和音乐而造福社会。这种新的理解也可能导致音乐教育的新技术。这项工作的另一个应用是在博物馆创造互动展品，人们可以在那里指挥虚拟管弦乐队，也许有一天人们会喜欢在家里指挥个人电子管弦乐队。手势传感的进展在医疗保健领域具有巨大的潜力。使用人体运动来检测活动的水平和类型以及情绪状态可能会带来巨大的好处，使计算机能够监控受伤、高危患者和老年人的恢复情况。

Music conducting is a sophisticated form of gestural communication. Despite centuries of practice, there is no consensus regarding even what is communicated, much less how the information is encoded and decoded. This project develops simultaneous advances in the nature of conducting through instrumentation and measurement; and in the techniques for sensing, detection, and interpretation of human gesture. The project records conducting gestures using wearable, wireless sensors and synchronizes gestural data with video and audio. Working closely with musicians and conductors, the project will develop new hypotheses about conducting gestures, develop new sensors to capture relevant information, and then use machine learning to create a conducting gesture recognition system. The results are a characterization of the information conveyed by conductors and an improved generation of wearable sensors and techniques for sensing human motion.The broader impact of this work includes expanding the role of experimental science in the arts, especially music. Through work with professional and student orchestras, the methods and benefits of research will be clearly communicated to non-scientists. This work will also benefit society through a better understanding of conducting and music. This new understanding may also lead to new techniques for music education. Another application of this work is the creation of interactive exhibits at museums where people can conduct a virtual orchestra, and perhaps people will someday enjoy conducting a personal electronic orchestra at home. Advances in gestural sensing have enormous potential in the field of health care. Using human motion to detect levels and types of activity as well as emotional state could offer great benefits, allowing computers to monitor recovery from injury, at-risk patients, and the elderly.