Representations and Interfaces for storytelling with 3D animated characters

点击翻译按钮获取中文摘要

Representations and Interfaces for storytelling with animated characters Across culture and time, storytelling with anthropomorphic characters lies at the very core of the human spirit. Humans have evolved with stories and storytelling in turn has evolved with humans. The creation and sharing of stories, through text, audio, illustration, animation, image, or video, is all increasingly digital. Such creative digital content has a pivotal role in shaping and connecting humanity. The digital platform has unprecedentedly democratized self-expression, publishing, recording, and sharing of stories through blogs, images, audio and video. Animation as a medium for storytelling however, remains beyond the reach of most creative people. This proposal will research digital representations, tools and interfaces for the creation of stories involving animated characters. Aspects of animation are getting easier: assets are readily available online (eg. Sketchfab, Metahuman); character motion can be similarly accessed (eg. Mixamo); facial animation can be driven by speech or video (eg. JALI, faceware); and these scenes can be assembled in animation software and game engines (eg. Maya, Adobe Char. animator, Unreal). Credible animation however, requires a space-time integrity not realized by naively assembling disparate assets. Even today, storytellers and directors at animation studios execute creative visions frame-to-frame, through teams of skilled animators. The non-linear temporal pacing and causality of events in story-time however, is not inherently captured or preserved as a linear sequence of frames. While data-driven techniques can learn space-time co-relations in animation, their latent spaces do not provide users' with meaningful manipulation handles. This research will thus focus on performance driven storytelling, structured into three themes. Face/body animation representations: Faces are typically represented as a linear combination of FACS-based shapes, and bodies as joint skeletons. The onus of maintaining the complex co-relations between various face/body parameters, in static pose and animation, is difficult. Representations, such as staggered pose manifolds, will be explored, to ease user control of animated characters. Space-time animation frameworks: Stories can have a complex, branching non-linear space-time structure. This theme will conduct research into animation techniques that eschew the traditional linear time slider, in favour of stacked storyboards, layered animation behaviour, and audience-driven story abstraction and re-timing.   Character animation interfaces: This theme will research a variety of novel interfaces, ranging from facial doodles, latent space animation layouts, animation curve handles, transcript and speech-driven storyboards, and direct 3D motion control in XR. Animated character storytelling as addressed by this proposal is critically important, as digital humans become extensions of physical lives.