Representations and Interfaces for storytelling with 3D animated characters

跨越文化和时代，用拟人化角色讲故事是人类精神的核心。人类随着故事而进化，而讲故事也随着人类而进化。通过文本、音频、插图、动画、图像或视频来创作和分享故事，这些都越来越数字化。这种创造性的数字内容在塑造和连接人类方面发挥着关键作用。数字平台通过博客、图像、音频和视频实现了前所未有的自我表达、出版、记录和分享故事的民主化。然而，动画作为一种讲故事的媒介，仍然超出了大多数创意人士的能力范围。该提案将研究涉及动画人物的故事创作的数字表示、工具和界面。动画的各个方面变得越来越容易：资产随时可以在网上获得(例如。Sketchfab Metahuman);角色运动也可以类似地访问(例如：Mixamo);面部动画可以由语音或视频驱动。JALI faceware);这些场景可以在动画软件和游戏引擎(如。Maya, Adobe Char。动画师,不真实)。然而，可靠的动画需要时空的完整性，而不是通过天真地组装不同的资产来实现的。即使在今天，动画工作室的故事讲述者和导演也通过熟练的动画师团队逐帧执行创意愿景。然而，故事时间中事件的非线性时间节奏和因果关系并不会被捕捉或保存为线性帧序列。虽然数据驱动技术可以学习动画中的时空相互关系，但其潜在空间并不能为用户提供有意义的操作手柄。因此，本研究将侧重于绩效驱动的故事叙述，分为三个主题。面部/身体动画表示：面部通常表示为基于facs形状的线性组合，而身体则表示为关节骨架。在静态姿态和动画中，维护各种面部/身体参数之间复杂的相互关系是一项艰巨的任务。表示，如交错姿势流形，将探讨，以方便用户控制动画人物。时空动画框架：故事可能具有复杂的、分支的非线性时空结构。本主题将对动画技术进行研究，避免传统的线性时间滑块，支持堆叠故事板，分层动画行为，以及观众驱动的故事抽象和重新计时。角色动画界面：本主题将研究各种新颖的界面，包括面部涂鸦，潜在空间动画布局，动画曲线处理，文字和语音驱动的故事板，以及XR中的直接3D运动控制。随着数字人类成为现实生活的延伸，这一提议所解决的动画角色叙事至关重要。

Representations and Interfaces for storytelling with animated characters Across culture and time, storytelling with anthropomorphic characters lies at the very core of the human spirit. Humans have evolved with stories and storytelling in turn has evolved with humans. The creation and sharing of stories, through text, audio, illustration, animation, image, or video, is all increasingly digital. Such creative digital content has a pivotal role in shaping and connecting humanity. The digital platform has unprecedentedly democratized self-expression, publishing, recording, and sharing of stories through blogs, images, audio and video. Animation as a medium for storytelling however, remains beyond the reach of most creative people. This proposal will research digital representations, tools and interfaces for the creation of stories involving animated characters. Aspects of animation are getting easier: assets are readily available online (eg. Sketchfab, Metahuman); character motion can be similarly accessed (eg. Mixamo); facial animation can be driven by speech or video (eg. JALI, faceware); and these scenes can be assembled in animation software and game engines (eg. Maya, Adobe Char. animator, Unreal). Credible animation however, requires a space-time integrity not realized by naively assembling disparate assets. Even today, storytellers and directors at animation studios execute creative visions frame-to-frame, through teams of skilled animators. The non-linear temporal pacing and causality of events in story-time however, is not inherently captured or preserved as a linear sequence of frames. While data-driven techniques can learn space-time co-relations in animation, their latent spaces do not provide users' with meaningful manipulation handles. This research will thus focus on performance driven storytelling, structured into three themes. Face/body animation representations: Faces are typically represented as a linear combination of FACS-based shapes, and bodies as joint skeletons. The onus of maintaining the complex co-relations between various face/body parameters, in static pose and animation, is difficult. Representations, such as staggered pose manifolds, will be explored, to ease user control of animated characters. Space-time animation frameworks: Stories can have a complex, branching non-linear space-time structure. This theme will conduct research into animation techniques that eschew the traditional linear time slider, in favour of stacked storyboards, layered animation behaviour, and audience-driven story abstraction and re-timing.   Character animation interfaces: This theme will research a variety of novel interfaces, ranging from facial doodles, latent space animation layouts, animation curve handles, transcript and speech-driven storyboards, and direct 3D motion control in XR. Animated character storytelling as addressed by this proposal is critically important, as digital humans become extensions of physical lives.