Time Information for Multimedia Processing and Analysis

数字多媒体数据量每天都在以惊人的速度增长。多媒体研究的主要挑战之一是开发算法，从这些海量的多媒体信号中提取信息，并提供更有效的工具来与它们交互。使用数字信号处理和机器学习的各种技术，现在可以分析数以百万计的图像、视频和歌曲。最近的另一个趋势是使用丰富的表现力方式与计算机交互，超越了传统的屏幕/键盘/鼠标范例。这些包括触摸屏上的手势，以及基于各种传感器的全身跟踪界面，如加速计、麦克风阵列和使用结构光的深度相机。结合由丰富的多模式输入控制的大规模多媒体档案分析的工作的例子很少。将传感器、执行器、计算机和人类结合在一起的互动音乐表演是少数几个产生这种协同效应的领域之一。 建立这类系统具有挑战性，因为它不仅需要跨学科的专门知识，而且大多数现有的方案编制环境只解决出现的一些问题。它们也缺乏表示时间和概率的通用抽象，而时间和概率是构建此类交互系统的基础。我们的主要目标是形式化、设计和实现更好的工具、语言和框架，以构建涉及大规模机器学习和丰富用户输入的交互式多媒体工具。时间建模是交互式多媒体的基础，我们的目标是探索时间如何变得更加明确，并成为用于编程的抽象的一部分。我们计划将统计、机器学习、信号处理和数据流编程等概念从软件库中的任意约定提升到编程实践中的一流原语。

The amount of multimedia data available digitally is growing daily at an astounding rate. One of the main challenges in multimedia research is to develop algorithms to extract information from these vast collections of multimedia signals and provide more effective tools for interacting with them. Using a variety of techniques from digital signal processing and machine learning it is now possible to analyze millions of images, videos and songs. Another recent trend is the use of rich expressive ways of interacting with computers moving beyond the traditional screen/keyboard/mouse paradigm. These include gestures on touchscreens as well as full body tracking interfaces based on a variety of sensors such as accelerometers, microphone arrays and depth cameras using structured light. There are few examples of work that combine large scale analysis of multimedia archives controlled by rich multi-modal input. Interactive music performances that combine sensors, actuators, computers and humans are one of the few areas where such synergies are taking place. Building such systems is challenging as not only it requires interdisciplinary expertise but the majority of existing programming environments address only some of the issues that arise. They also lack generic abstractions for expressing time and probabilities which are fundamental in building such interactive systems. Our main objective is to formalize, design and implement better tools, languages and frameworks for building interactive multimedia tools that involve both large-scale machine learning and rich user input. Modeling time is fundamental to interactive multimedia and our goal is to explore how time can become more explicit and part of the abstractions used to program. We plan to elevate concepts from statistics, machine learning, signal processing and data-flow programming from arbitrary conventions in a software library to first class primitives in programming practice.