RI: Small: Lightly Supervised Deep Learning for Multi-Frame Visual Motion Analysis

该项目解决了确定视频序列中每个时间点的每个图像点的运动的基本视频分析问题。虽然这个问题对人类来说似乎毫不费力，但对计算机来说仍然是一个挑战，特别是当物体快速移动，数量庞大或以复杂的方式移动时。在过去的几十年里，计算机视觉领域在这个问题上取得了巨大的进步，但仍然有很大的改进空间。该项目利用机器学习的最新发展来提高视频中点运动估计的准确性和可靠性。除了研究生，该项目还将涉及杜克大学巴斯连接项目主持下的本科生。该项目面向大学一年级的学生。该项目的重点包括开发合适的运动表示、设计和训练深度学习架构以及性能评估。代表性挑战至关重要：虽然两个帧之间的点的运动是连接运动起点和终点的简单矢量，但当涉及多个帧时，它就变成了轨迹。附近点的轨迹在属于同一物体时通常是相似的，但在不同物体上时则不同，这一推动力将发展出相应的分段连续轨迹场的数学。深度学习架构和相应的学习算法，作为第二个重点，将被重新考虑，以最大限度地利用不同点和时间的运动之间的关系。最后，性能评估将提供对正在开发的算法的权衡、优点和缺点的细致入微的理解，并将帮助确定哪些方法和参数设置最适合哪种类型的视频。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

This project addresses the fundamental video analysis problem of determining the motion of every image point at every point in time in a video sequence. While apparently effortless for people, this problem is still a challenge for computers, especially when objects move fast, in large numbers, or in complex ways. The field of computer vision has made tremendous strides on this problem in the last few decades, but there is still ample room for improvement. This project draws on recent developments in machine learning to improve the accuracy and reliability of the estimates of point motion in video. In addition to graduate students, the project will involve undergraduates under the auspices of the Bass Connections program at Duke University. This program reaches out to students in their first college years.The thrusts of the project include the development of suitable representations of motion, the design and training of deep learning architectures, and performance evaluation. The representational challenge is paramount: While the motion of a point between two frames is a simple vector connecting the start and end point of the motion, it becomes a trajectory when multiple frames are involved. Trajectories of nearby points are often similar when they belong to the same object, but they are different when they are on different objects, and this thrust will develop the mathematics for the piecewise continuous fields of trajectories that arise as a result. Deep learning architectures and corresponding learning algorithms, at the center of the second thrust, will be re-thought to take best advantage of relations between motions at different points and times. Finally, performance evaluation will provide a nuanced understanding of the trade-offs, strengths, and weaknesses of the algorithms being developed, and will help determine what methods and parameter settings work best for what type of video.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.