Representations and Interfaces for storytelling with 3D animated characters

动画角色讲故事的表现和界面跨越文化和时间，拟人化角色讲故事是人类精神的核心。人类随着故事而进化，而讲故事也随着人类而进化。通过文本、音频、插图、动画、图像或视频创作和分享故事，都越来越数字化。这种创造性的数字内容在塑造和连接人类方面发挥着关键作用。数字平台使自我表达、出版、记录和通过博客、图像、音频和视频分享故事的方式空前民主化。然而，动画作为一种讲故事的媒介，仍然超出了大多数有创造力的人的能力。该提案将研究数字表示，工具和界面，用于创建涉及动画角色的故事。动画的各个方面变得越来越容易：资源可以在网上轻松获取（例如。Sketchfab，超人类）;角色动作可以类似地访问（例如。Mixamo）;面部动画可以由语音或视频驱动（例如，这些场景可以在动画软件和游戏引擎中组装（如JALI，Faceware）。Maya、Adobe Char。动画师，虚幻）。然而，可信的动画需要一个时空完整性，而不是通过天真地组装不同的资产来实现。即使在今天，动画工作室的故事讲述者和导演也通过熟练的动画师团队逐帧地执行创造性的愿景。然而，故事时间中事件的非线性时间节奏和因果关系并不固有地被捕获或保存为线性帧序列。虽然数据驱动技术可以学习动画中的时空相关性，但它们的潜在空间并不能为用户提供有意义的操纵手柄。因此，本研究将侧重于表演驱动的故事，分为三个主题。面/身体动画表示：面通常表示为基于FACS的形状的线性组合，身体表示为关节骨架。在静态姿势和动画中，维持各种面部/身体参数之间的复杂相互关系的责任是困难的。表示，如交错构成流形，将探讨，以减轻用户控制的动画角色。时空动画框架：故事可以有一个复杂的、分支的非线性时空结构。这个主题将进行动画技术的研究，避开传统的线性时间滑块，有利于堆叠故事板，分层动画行为，观众驱动的故事抽象和重新定时。 角色动画界面：本主题将研究各种新颖的界面，包括面部涂鸦，潜在空间动画布局，动画曲线手柄，转录和语音驱动的故事板，以及XR中的直接3D运动控制。这个提议所涉及的动画角色讲故事是至关重要的，因为数字人类成为物理生命的延伸。

Representations and Interfaces for storytelling with animated characters Across culture and time, storytelling with anthropomorphic characters lies at the very core of the human spirit. Humans have evolved with stories and storytelling in turn has evolved with humans. The creation and sharing of stories, through text, audio, illustration, animation, image, or video, is all increasingly digital. Such creative digital content has a pivotal role in shaping and connecting humanity. The digital platform has unprecedentedly democratized self-expression, publishing, recording, and sharing of stories through blogs, images, audio and video. Animation as a medium for storytelling however, remains beyond the reach of most creative people. This proposal will research digital representations, tools and interfaces for the creation of stories involving animated characters. Aspects of animation are getting easier: assets are readily available online (eg. Sketchfab, Metahuman); character motion can be similarly accessed (eg. Mixamo); facial animation can be driven by speech or video (eg. JALI, faceware); and these scenes can be assembled in animation software and game engines (eg. Maya, Adobe Char. animator, Unreal). Credible animation however, requires a space-time integrity not realized by naively assembling disparate assets. Even today, storytellers and directors at animation studios execute creative visions frame-to-frame, through teams of skilled animators. The non-linear temporal pacing and causality of events in story-time however, is not inherently captured or preserved as a linear sequence of frames. While data-driven techniques can learn space-time co-relations in animation, their latent spaces do not provide users' with meaningful manipulation handles. This research will thus focus on performance driven storytelling, structured into three themes. Face/body animation representations: Faces are typically represented as a linear combination of FACS-based shapes, and bodies as joint skeletons. The onus of maintaining the complex co-relations between various face/body parameters, in static pose and animation, is difficult. Representations, such as staggered pose manifolds, will be explored, to ease user control of animated characters. Space-time animation frameworks: Stories can have a complex, branching non-linear space-time structure. This theme will conduct research into animation techniques that eschew the traditional linear time slider, in favour of stacked storyboards, layered animation behaviour, and audience-driven story abstraction and re-timing.   Character animation interfaces: This theme will research a variety of novel interfaces, ranging from facial doodles, latent space animation layouts, animation curve handles, transcript and speech-driven storyboards, and direct 3D motion control in XR. Animated character storytelling as addressed by this proposal is critically important, as digital humans become extensions of physical lives.