A Pedagogical Workflow for Interconnected Learning for Animation, Visual Effects, and Game Design Students

Published at SIGGRAPH Asia 2024 Educator’s Forum, Tokyo International Forum, Japan. https://dl.acm.org/doi/10.1145/3680533.3697059

Written by: Hossein Najafi, Jason Kennedy, Gregory Bennett, Masaya Todoroki and Emily Ramsay

ABSTRACT

The paper presents a case study in a curriculum structure that aims to integrate specialised course content across a range of primary study pathways. The 3-year Bachelor of Design degree at the School of Art and Design, Auckland University of Technology, New Zealand, offers a Major in Animation, Visual Effects and Game Design in which students can follow specialised pathways of study in these three areas. In addition to their selected major, students can select two minors to support their major course of study. A popular choice is the Motion Capture Minor that has strong technical and creative relevance to all three study pathways, and due to time and scope restraints is not able to be covered directly in the curriculum content of the majors. This paper examines this major/minor structure as a useful interconnected curriculum framework that supports a learning environment whereby students can gain both in-depth creative and technical expertise in a specialised and future-focussed creative technology, but also bring that knowledge to their major pathways of study.

KEYWORDS

education, curriculum design, motion capture, performance capture

1 - INTRODUCTION

The equipment available in a motion capture facility largely determines the type and scope of projects that can be produced therein. By extension, this also influences the nature of cross-disciplinary engagement that can be achieved. This paper describes interconnected learning outcomes related to the curriculum infrastructure offered in the Bachelor of Design at Auckland University of Technology. The AUT Motion Capture Lab system is comprised of 24 Motion Analysis Raptor-4 cameras used for passive optical body tracking. The Motion Analysis system records at 200 frames-per-second (fps) with a resolution of 2352×1728 pixels and can accommodate up to six simultaneous performers within the studio’s 6×6×3 metre capture volume. Facial performance is captured using the Faceware ProHD system, which includes a monocular head-mounted camera that records RGB video at 59.94 fps and 720×1280 pixels. Additionally, the facility includes a photogrammetry setup composed of DSLR cameras on tripods at various heights and orientations around a subject. The choice and deployment of equipment within the AUT Motion Capture Lab is largely determined by two goals: accommodating the pedagogical requirements of the university’s courses (in particular, the motion capture minor available to all AUT students); and functioning as an education hub to facilitate collaboration with global partners in education and industry, while ensuring professional currency.

The AUT motion capture minor is delivered over four semesters. In addition to teaching practical and technical foundations for recording, cleaning, and retargeting performance data, the minor emphasises creative means for visualising human movement, from precise realism to artistic abstraction. The motion capture minor is consistently one of the top two most popular minors with the School of Art & Design (based on the number of students enrolled in the course). In various years, professional productions have partnered with the AUT Motion Capture Lab to produce their work, both in the studio and on location; students within the motion capture minor often elect to participate in recording and cleaning this data. Students graduating from the motion capture minor have practical and creative advantages when applying for work in related fields; for example, many of the AUT graduates hired by visual effects company Wētā FX (Wellington, New Zealand) completed the motion capture minor.

Motion capture (MoCap) serves as a nexus between live-action and animated performance. For an actor, MoCap provides opportunities for developing a dramatic relationship with a virtual actor (vactor) counterpart. This includes how the vactor’s design affects the actor’s performance, as well as how an actor understands the extent of their contribution to the final on-screen performance. For animators, MoCap provides opportunities for better seeing, understanding, and analysing movement through both mechanical and dramatic lenses. Animators who “suit up” to record their own MoCap performances develop a stronger embodied connection between intention, physicality, and the resulting digitised action [Kennedy 2021]. Johnson-Glenberg and Megowan-Romanowicz [2017] highlight various forms of enhanced engagement motion capture provides within an educational context, including embodied learning within interactive environments and deeper involvement within a given narrative.

The two examples above demonstrate the potential for motion capture to expand beyond traditional delivery models for classroom learning. Indeed, a well-supported motion capture facility can serve as an education hub to bring together pedagogical strategies across a wide range of educational activities. Higher-education scholar Jane Knight [2011] describes an education hub as “a planned effort to build a critical mass of local and international actors strategically engaged in cross-border education, training, knowledge production, and innovation initiatives”. Here, the term “actor” is used to describe any entity contributing to the hub, including, but not limited to, students, educational institutions, companies, and technology centres [Knight 2013]. This understanding allows education hubs to emerge globally across a range of scales (such as countries, zones, cities, or communities) with a diversity of actors specific to the regions and facilities in question. Connections between actors provide opportunities for education as well as professional services, bringing together a range of backgrounds to contribute to projects that may not otherwise be available in any single location. This connection between various actors creates opportunities for cross-disciplinary engagement. Notably, a project/activity can benefit from a range of creative/professional backgrounds not available in a single location; in so doing, an education hub can support initiatives that may not be feasible otherwise.

The AUT Motion Capture Lab provides examples of these strategies: MoCap educators share their knowledge with film production teams that may not have the experience/facilities to produce MoCap on their own; in turn, film production teams provide valuable on-set experience for motion capture students seeking entry into the field. This paper describes a further range of collaborative outcomes accomplished through the AUT Motion Capture Lab in its role as an education hub.

2 - MOTION CAPTURE AS A PEDAGOGICAL TOOL

The integration of motion MoCap technology in higher education offers transformative potential for teaching and learning across disciplines such as animation, visual effects (VFX), and game design. As Bennett, Najafi, and Jackson [2023] highlight, the introduction of virtual production tools, including MoCap, facilitates an adaptive and responsive approach to curriculum development, ensuring alignment with industry standards and preparing students for future challenges in the creative industries. The provided diagram (see Figure 1) illustrates the central role of the MoCap Minor at AUT, acting as a nexus point interconnecting the three pathways within the Animation, Visual Effects, and Game Design major. This interconnected structure emphasizes how MoCap serves as a bridge between these disciplines, allowing students to gain a comprehensive understanding of each field's intricacies. The MoCap minor is not just an isolated program; it serves as a pivotal component that integrates knowledge from the pathways within the major, providing students with a multidisciplinary educational experience. Furthermore, the diagram includes an XR (extended reality) minor, indicating potential cross-disciplinary collaboration and innovation, particularly in areas involving immersive technologies. Motion capture serves as a crucial bridge between theoretical concepts and practical applications. By engaging with MoCap technology, students can directly experience the intricacies of the animation process, thereby enhancing their technical competencies and creative problem-solving skills [Bennett and Kruse 2015]. The utilisation of MoCap allows students to experiment with various aspects of animation and visual effects, encouraging innovation and critical thinking in a controlled educational setting.

The pedagogical benefits of MoCap extend to the development of technical skills essential for professional practice. By working with MoCap, students gain proficiency in capturing, cleaning, and editing performance data, which are crucial skills in the animation and VFX industries. This experiential learning approach not only prepares students for real-world challenges but also enhances their employability upon graduation. Furthermore, motion capture technology fosters interdisciplinary collaboration, a vital component of contemporary educational strategies. The ability to integrate MoCap into various disciplines encourages students from different fields to work together, facilitating a multidisciplinary approach to problem-solving. This collaboration enhances the learning experience by providing students with diverse perspectives and expertise, ultimately leading to more holistic educational outcomes. Beyond technical skills, motion capture serves as a tool for creative exploration. By providing students with the freedom to experiment with MoCap data, educators can nurture a culture of creativity and innovation. The immersive nature of MoCap technology allows students to engage deeply with narrative storytelling, enabling them to create realistic digital characters and environments that resonate with audiences. This creative exploration is vital for developing the next generation of animators and VFX artists. The motion capture minor at AUT exemplifies the successful integration of MoCap technology in an academic setting. The minor, which consists of four courses spread across three years, provides students with a comprehensive understanding of both body and facial motion capture, as well as virtual camera operation [Bennett et al. 2023]. This curriculum is designed to equip students with the skills necessary to thrive in the competitive animation and VFX industries. The program's emphasis on practical application is reflected in the diverse range of projects students undertake, from creating digital character performances to developing complex previsualisation sequences. By engaging with MoCap technology, students at AUT produce high-quality work that demonstrates both technical proficiency and creative innovation, preparing them for successful careers in their chosen fields.

2.1 Motion Capture as a Pedagogical Facilitator for Animation Students

The inclusion of motion capture within a character animation curriculum can enhance students’ understanding of human movement, performance dynamics, and the nuanced interplay between live-action and animated performance. By engaging students with MoCap practice, educators can create stronger linkages between theory and practice: students apply theoretical knowledge in real-world contexts to enhance their technical competencies and creative problem-solving skills [Bennett and Kruse 2015].

Within the AUT curriculum, MoCap technology has significantly enhanced students' understanding of animation techniques by providing real-time interfaces for capturing and analysing motion. This technology allows students to experiment with various character movements, gaining insights into the biomechanics of motion and the principles of animation. The ability to capture live-action movement and retarget it as digital performance results in a deeper understanding of the relationship between character and narrative.

MoCap also allows students to engage with complex performance techniques that are challenging to master through traditional animation methods. For instance, the integration of facial motion capture technology (i.e.: performance capture) enables students to record nuanced human expressions, which ads emotional depth and believability to animated characters. This capability is crucial when developing animation for more realistic character designs, and results in more sophisticated performances that resonate with audiences and demonstrate a high level of artistic and technical skill.

Case Study: “Paper Cuts” (Ettiene de Jager)

Figure 2: Ettiene de Jager's Motion Capture Study

The student animated short film “Paper Cuts” [de Jager 2024] exemplifies the transformative potential of MoCap technology in animation education, allowing for the technology to serve as an education hub that supports students’ capabilities beyond just the skills learned in the curriculum. De Jager developed a series of stylistic animations that explore the intricacies of character performance within an ostensibly analogue context (Figure 2). In this animation, de Jager taps into his fencing experience to produce a complex animation full of character and prop interactions that would be lengthy and difficult to manually animate. He created a custom toon-style shading network in Autodesk Maya to render the characters in the look of hand-drawn animation (with additional visuals created within Adobe After Effects) (Figure 3). By interpreting the MoCap data through this stylistic strategy, de Jager maintains consistency of form and believable motion while masking the digital trace of the tracked movement. By leveraging the capabilities of MoCap technology, de Jager produced an animation that exhibits a high degree of fluidity, emulates a 2D animation aesthetic, and maintains the authenticity of his designs. This fusion of practices yields a personal and distinct final output. This approach not only broadens the scope of animation possibilities, but also paves the way for future experimental endeavours.

Figure 3: scene setup for “Paper Cuts” within the Autodesk Maya interface.
 
Several papers throughout the Animation, Visual Effects, and Game Design curriculum at AUT allow students to engage with motion capture in their projects. This fosters a collaborative learning environment with comprehensive MoCap skills, resulting in creative and diverse outcomes. Students are encouraged to creatively and technically experiment with the MoCap tools, thereby expanding their understanding of motion capture within the animation pipeline. They are offered the freedom and flexibility to explore potential uses for MoCap data while working on assignment briefs that prepare them for work in the industry and academic research.

There are several ways animation education can benefit from a motion capture laboratory serving as an education hub:

• Realistic character animation: MoCap records the movements associated with intricate human motions, resulting in a transference of enacted behaviour between flesh-and-blood performers and their digital counterparts. When done well, this transference enhances the realism and believability of animated character performances, with the goal of producing greater emotional engagement. This technique enables students to bridge the gap between lived experience and animation by crafting characters that exhibit high degrees of detail and believability.
• Reducing animation time: in specific situations, MoCap may reduce animation time and labour when compared to manual keyframing. This efficiency is most noticeable when students work on their own to produce animations on tight deadlines. This works best when more value is placed on creative visualisations than strict adherence to a source performance. With MoCap, animators can quickly block out complex scenes, freeing up time for refining and enhancing their work, which is crucial in a fast-paced educational environment.
• Facial animation: performance capture (PeCap) technology enables animators to record and retarget precise expressive facial movements. By incorporating PeCap into their workflow, students can create nuanced and lifelike expressions, enhancing storytelling and audience connection in character-driven narratives.
• Multi-sensory learning: MoCap offers a dynamic learning experience that engages students through a range of sensory inputs; in turn, this enhances their understanding and application of animation principles. By allowing students to interact with motion data, MoCap fosters a deeper grasp of movement dynamics and encourages hands-on exploration. This approach supports active learning and creativity, enables students to experiment with animation in innovative ways, and ultimately enriches the educational process by better preparing them for animation careers.

2.2 Motion Capture as a Pedagogical Facilitator for Game Design Students

The game design pathway encompasses a diverse array of assignments, spanning both technical and artistic domains. During the initial two semesters, the curriculum emphasizes the enhancement of students’ skills in asset creation (2D/3D), coding and understanding of game theory. Rather than just focusing on a singular aspect of game design, this holistic approach equips students with a broad skill set. By employing a methodology akin to game jams, students are encouraged to rapidly prototype their ideas and learn from their mistakes, thereby fostering a deeper understanding of game development concepts. This pedagogical strategy not only facilitates the testing of ideas but also cultivates resilience and adaptability through the concept of “graceful failure,” wherein learning from errors is an integral component of the educational process [Plass et al., 2015]. The skills and insights acquired through this approach are subsequently applied to the second-year multiplayer game group project and the final-year capstone project. In these projects, students can specialise in specific areas of interest, thereby refining their expertise and contributing to a collaborative development environment.

Game development is a multifaceted discipline, with 3D animation representing a small yet time-consuming component. Moreover, throughout various projects, students may prefer to concentrate on learning game design aspects beyond 3D animation. The motion capture studio addresses this challenge by enabling students to efficiently generate animations for their projects. Although students can source motion data from platforms such as the Unity Asset Store, Unreal Marketplace, and Adobe’s Mixamo animation library, they frequently encounter the need for specific actions that are not available pre-made, such as interactions between multiple characters. Furthermore, access to raw motion data files allows students to learn the intricacies of cleaning and manipulating data to meet the requirements of game engines, thereby enhancing their technical proficiency and understanding of the animation process.

Character animation workflows in game development are varied and often influenced by game engine requirements, necessitating dynamic and responsive animations. For example, linking character animations to in-game movement can be managed either through coding or by employing root motion, where the animation itself drives the movement. Each approach has its advantages and impacts the fluidity of character movement and gameplay feel. Furthermore, students facing challenges with the coding aspects of game design have found using root motion to drive character movement an easier approach. Historically, our students specialising in motion capture have collected their own motion capture data for their games, thereby exploring both approaches and developing the skills necessary to transition between them. This experiential knowledge is now being incorporated into the multiplayer game assignment, emphasizing the importance of learning different animation techniques in game development.

Case Study: “Hide & Shriek” (Jessica Ryou, Pui Yee Kong, Hayley Son, and Vivian Chu)

For their second-year multiplayer game project titled Hide & Shriek, students Ryou et al. used motion capture (Figure 4) to animate both their characters and their creature (Figure 5). Hide & Shriek is a 4 vs. 1 online multiplayer survival horror game inspired by Dead by Daylight [Behaviour Interactive 2016] and other similar titles.

Figure 4: second-year student, Nick Johnson, performing in a MoCap suit for the creature animation.

As this was their inaugural online multiplayer project, the team required extensive research and development of multiplayer systems. Using MoCap, they were able to efficiently test and implement animations that were tailored to their game, thereby eliminating the need to source similar motions or manually keyframe the animations. This technological efficiency afforded the team the opportunity to allocate more time to other critical aspects of the project, such as refining gameplay mechanics and conducting research into additional areas of game development.

Figure 5: an in-software preview of the creature rig within Autodesk Maya.

2.3 Motion Capture as a Pedagogical Facilitator for Visual Effects Students

The VFX pathway at AUT consists of four major assignments over three years of study: a music video, “The Creature”, a VFX animated documentary, and a capstone project. In all these assignments, motion capture plays a significant role. Although MoCap is not an official part of the curriculum of the AVG major, it leverages many elements from students’ knowledge gained within the MoCap minor. AUT’s MoCap Lab and MoCap minor act as a hub that serves different pathways, including VFX, where students bring elements from MoCap assignments into their VFX projects.

For instance, the final capstone courses emphasise the importance of a well-researched, well-developed seminal project that includes storyboarding, animatics, modelling, texturing, rigging, animating, 3D layout, and prototyping. By incorporating motion capture workshops, students gain hands-on experience, learning how to set up, calibrate, and process MoCap data for digital creatures within their VFX projects. This significantly impacts the quality of their outcomes in the animation and performance stages. As Harvey [2010] propose, practical exposure helps students grasp complex performances more effectively and develop technical skills directly applicable in the VFX industry, such as rigging, animation, and storytelling.

Moreover, project-based learning, where students undertake MoCap-based projects (like animating a CG creature or developing a short film with MoCap characters), allows them to apply their knowledge in real-world scenarios. Such projects also foster collaboration and teamwork, as seen in “The Creature” assignment, where students work in teams to create a VFX-based short film focused on integrating a CG creature within live-action; this requires students to engage in all stages from preproduction through postproduction. Encouraging interdisciplinary collaboration between VFX students and those from other disciplines, such as acting or dance, enriches the motion capture experience and results in more realistic/believable animations.

Many VFX students from AUT secure employment at renowned studios like Wētā FX, where they apply their combined knowledge of VFX and motion capture to high-level industry projects. This integration of skills not only enhances their employability but also enables them to contribute significantly to major productions. Partnerships with industry professionals and companies provide students with access to the latest motion capture technology and real-world insights, preparing them for successful careers in the VFX field.

Case Study: “Bucket List” (Anya Hill and Emily Liu)

An illustrative case study is the short film Bucket List, created by VFX students Hill and Liu [2023], who utilise MoCap technology in their heart-warming capstone project to create a digital bear character that remember their friend who passed away years ago (Figure 6).

Figure 6: a snapshot from the short film Bucket List. The bear represents a departed friend. Hill and Liu extensively used MoCap to animate the bear character.

In this project, Anya Hill performed the motion capture for the character bear (Figure 7).

Figure 7: Final year student, Anya Hill acting in MoCap suit.

Within this project, there are multiple shots involving live-action compositing and physics-based simulations. MoCap enabled the team to attain an acceptable level of realistic motion in their VFX shots. For example, in a shot that the bear jumps into the pool alongside her friends (Figure 8), MoCap data was critical in achieving a believable splash simulation in Houdini software that syncs with the live-action footage.

Figure 7: the use of MoCap data in Houdini simulations.

3 CONCLUSION AND FUTURE DIRECTIONS

In conclusion, in this study, the integration of MoCap technology into higher education represents a transformative shift in pedagogical approaches for students in animation, visual effects, and game design. By acting as a pedagogical hub, the AUT Motion Capture Lab facilitates experiential learning, interdisciplinary collaboration, and innovation across various fields. The future of MoCap in education should focus on expanding its application to other disciplines, fostering collaborations with industry leaders, and establishing long-term goals that position MoCap as a cornerstone of modern educational practices. Collaborative efforts with technology companies, film studios, and research institutions enable educational institutions to explore new and innovative applications of MoCap, aligning educational programs with evolving industry demands [Knight, 2013]. Such partnerships not only prepare students for successful careers in dynamic and rapidly changing industries, but also empower them to become leaders and innovators in their respective fields.

We note that the implementation of motion capture technology in education provides a practical and immersive platform for students to engage with course material actively. This engagement leads to heightened motivation and a deeper understanding of complex concepts, as students can directly observe the real-time impact of their work in applied settings.

Also, we note that Mocap technology enhances the quality and complexity of student projects, allowing for more creative and innovative outcomes. Like Von Kagaoan (2017) we also realised the effectiveness of MoCap in translating acted performances into digital characters with differing physiologies, creating realistic and engaging digital content. This capability illustrates the transformative potential of MoCap in creating nuanced digital characters that resonate with audiences.

References

Behaviour Interactive. 2016. Dead by Daylight. Video game. Montreal, Canada. Available at: https://store.steampowered.com/app/381210/Dead_by_Daylight/

Gregory Bennett and Jan Kruse. 2015. Teaching Visual Storytelling for Virtual Production Pipelines Incorporating Motion Capture and Visual Effects. In SIGGRAPH Asia 2015 Symposium on Education (SA '15). Association for Computing Machinery, New York, NY, USA, Article 4, 1–8. https://doi.org/10.1145/2818498.2818516

Gregory Bennett, Hossein Najafi and Lee Jackson. 2023. Pedagogical Strategies for Teaching Virtual Production pipelines. In SIGGRAPH Asia 2023 Educator's Forum (SA '23). Association for Computing Machinery, New York, NY, USA, Article 11, 1–10. https://doi.org/10.1145/3610540.3627010

Ettiene de Jager. 2024. Paper Cuts. Animated short film. Auckland University of Technology.

Harvey, Louise. 2010. Designing Efficiency: The Benefits of Pre-Visualisation in Film and Animation Teaching Programs. EDULEARN10 Proceedings (pp. 4533-4541). IATED.

Mina C. Johnson-Glenberg and Colleen Megowan-Romanowicz. 2017. Embodied Science and Mixed Reality: How Gesture and Motion Capture Affect Physics Education. Cognitive Research: Principles and Implications 2, 24, 1-28. https://doi.org/10.1186/s41235-017-0060-9

Von Kagaoan. 2017. Translating Acted Performance Between an Actor and a sSynthespian of Differing Physiology. Honours thesis. Auckland University of Technology. Available at https://wovenlancer.artstation.com/projects/VdBl8n


Jason A. Kennedy. 2021. Acting and Its Double: A Practice-Led Investigation of the Nature of Acting Within Performance Capture. Ph.D. Dissertation. Auckland University of Technology. https://hdl.handle.net/10292/13943

Jane Knight. 2011. Education Hubs: A Fad, a Brand, an Innovation? Journal of Studies in International Education 15, 3, 221-240. https://doi.org/10.1177/1028315311398046

Jane Knight. 2013. Education Hubs: International, Regional, and Local Dimensions of Scale and Scope. Comparative Education 49, 3, 374-387. https://doi.org/10.1080/03050068.2013.803783

Jan L. Plass, Bruce D. Homer, B. D. and Charles K. Kinzer. 2015. Foundations of Game-Based Learning. Educational Psychologist, 50, 4, 258-283. https://doi.org/10.1080/00461520.2015.1122533