Animating Humans from Static Images via an Entirely Image-Based Approach

图像/视频在人物动画中有着广阔的前景，完全基于图像/视频的方法可以让我们通过直接利用真实的图像/视频来实现高真实感。不幸的是，有效地利用真实世界的图像/视频并不是一项简单的任务，重建人类运动的3D任意视图非常困难。目前，基于图像/视频的渲染（IVBR）通过使用从3D扫描仪或多摄像头捕获或改编的通用人体几何图形来实现这一目标，这涉及到昂贵的资源。事实上，人类的大脑有一种强大的内在能力，可以想象静态物体的运动。给定一些人体运动的图像，我们可以很容易地通过在脑海中想象一个虚拟的运动来解释它们，而不需要任何几何信息。然而，现有的计算技术在很大程度上仍然无法实现这种能力。受这一观察结果的启发，拟议的研究旨在进行一次高度投机的冒险，探索使计算机具备这种能力的新技术。一般来说，我们将研究从静态图像中使人物活起来的可行性，允许从几个关键图像直接重建他们的任意运动视图，而不需要他们的几何模型。虽然我们这里的目标是人类，但所检查的方法可以适用于广泛的铰接/非铰接主题。它将超越目前主要为具有固定形状的对象设计的IVBR形式，并将致力于实现传统上只有借助于几何模型才能实现的目标。它可能导致一种与所有现有技术根本不同的替代方案。这项可行性研究将只集中在完全基于图像的方法中最基本的问题，即人类视图重建（VRH），即我们是否可以仅从几张静态图像中创建任意视点下的人类运动图像。为了在不失去一般性的情况下测试这个想法，我们实验中涉及的许多数据集将由计算机创建。一旦找到VRH的解决方案，它将为使用真实捕获的图像进行培训的进一步研究打开大门，并在后续项目中研究有关控制，数据组织和压缩，图像组成，头发和布料运动等其他重要问题。为了在短时间内完成可行性研究，我们设计了详细的研究路线。一个基于学习的方法将采取建立统计模型的图像序列的人体运动，通过训练从现有的例子。随后，这些模型将用于构建新的人体运动序列。虽然有许多潜在的方法可以提供有效的用户控制，但为了保持对VRH的关注，我们将采取最直接的控制策略，即使用选定数量的图像来指示演员在一段时间内的关键姿势。这类似于动画中广泛采用的关键帧控制策略。这项研究也有很强的商业潜力，在广泛的娱乐相关业务领域，如图像/视频编辑，电脑游戏，电影行业等。他们在英国有很大的影响力，并创造了可观的全球收入。它将由我们在滑稽科技和中国的工业联系积极参与。Cinesite是世界上最大的计算机视觉效果和后期制作公司之一，而滑稽技术公司为全计算机动画提供革命性的软件，并拥有世界各地的客户。他们认识到这项研究的潜在市场价值，并将通过咨询、评估和开发提供强有力的支持。滑稽将提供最新的动画软件发布为这项研究免费。

Images/videos have a promising future for figure animation, an entirely image/video-based approach would allow us to achieve high realism by directly utilising real images/videos. Unfortunately, making effective use of real world images/videos is not a simple task and it is very difficult to reconstruct 3D arbitrary views for human motion. Currently, the Image/Video Based Rendering(IVBR) achieves this by using either captured or adapted generic human geometry from 3D scanners or multi-cameras, which involves costly resources.In fact, the human brain has a strong in-built capacity to imagine motion from static objects. Given a few images of a human motion, we can easily interpret them by envisaging a virtual movement in our mind, without the need of any geometric information. However, existing computing technology still largely falls short of such a capability. Motivated by this observation, the proposed research is designed to take a highly speculative adventure which will explore novel techniques to equip computers with such an ability. Generally speaking, we shall look into the feasibility of making human characters alive from their static images, allowing arbitrary views of their movement to be directly reconstructed from a few key images without requiring their geometric models. While we target humans here, the methodology examined can be applicable to a broad range of articulated/non-articulated subjects. It will go beyond the current form of IVBR, which was mainly designed for objects with fixed shapes, and will aim to achieve what is traditionally feasible only with the assistance of geometric models. It could lead to an alternative that is fundamentally different from all current techniques.This feasibility study will concentrate only on the most fundamental issue of the entirely image based approach , which is the View Reconstruction for Humans (VRH), i.e. whether we can create images of a human movement under arbitrary viewpoints just from a few static images. To test the idea without losing generality, many datasets involved in our experiment will be created by computers. Once a solution to VRH is found, it will open the door to further investigation using real captured images for the training and also to work on other important issues concerning control, data organization & compression, image compositions, hair and cloth motion, etc., in follow-on projects. To allow for the completion of this feasibility study in a short period, we have designed a detailed research route. A learning-based approach will be taken to build statistical models for image sequences of human motion through training from existing examples. Subsequently, such models will be used to construct new sequences of human motion. While there are many potential ways to provide effective user controls, in order to stay focused on VRH, we shall take the most straightforward control strategy, which will use a selected number of images to indicate the key postures of the actor over the time. This is analogous to the key-frame control strategy that is widely adopted in animation.This research also has strong commercial potential in a broad range of entertainment-related businesses in areas such as image/video editing, computer games, the film industry, etc. They have a major presence in the UK and generate significant global income. It will be actively invovled by our industrial contacts at Antics Technologies and Cinesite. Cinesite is one of the largest companies in the production of computer visual effects and post production in the world, while Antics Technologies provides revolutionary software for full computer animation and has world-wide customers. They have recognized the potential market values of this research and will provide strong support through consultancy, evaluation and exploitation. Antics will provide their latest animation software release for this research at no cost.