ADELAIDE, SOUTH AUSTRALIA — For Marvel Studios’ latest project, Shang-Chi and the Legend of the Ten Rings, Rising Sun Pictures delivered nearly 300 visual effects shots. The project is the studio’s first film to premiere exclusively in theatres in more than two years and was directed by Destin Daniel Cretton.
RSP’s work centered on the creation of a vast, CG environment representing a village called Ta Lo and its jungle environment. The digital jungle serves as the backdrop for much of the latter part of the film and is the site of some of its most electrifying action.
“We were thrilled that Marvel Studios trusted us with the development of one of the film’s key assets,” says RSP managing director Tony Clark. “The enormity of the set posed a huge challenge for our team, and they responded with their best work.”
RSP worked under the direction of production VFX supervisor Christopher Townsend and VFX producer Damien Carr.
The Ta Lo set describes a swath of jungle spanning 75 square kilometers. Loosely based on locations in Southeast Asia, it is meant to appear real to audiences but not identifiable as any actual place. RSP artists, led by visual effects supervisor Malte Sarnes, designed the geography of the landscape, including lakes, mountains, ancient forests and rice paddies, populated it with plants and myriad other details, and integrated practical elements shot on-set in Sydney.
“We studied forested areas in Vietnam and Indonesia, and reproduced the variety of plant-life found there as well as how plants are distributed across the environment,” notes Sarnes. “We mimicked the light, humidity and feel of the jungle.”
Production of the Ta Lo environment occupied the team for months. Artists began by laying out the physical structure of the environment and then added successively finer layers of detail.
“We built a library of plants that were interesting and appropriate to the environment, including many varieties of bamboo,” recalls CG supervisor Julian Hutchens. “Once we had the proxy geometry in place, we began fleshing things out. We ran erosion simulations to wear away parts of the landscape where water would naturally run and carve out deep crevasses. Then we added species of trees and other plants, distributing them based on wet versus dry areas and the amount of sunlight.”
The complexity of a landscape so large presented numerous challenges.
“We developed a logic that allowed us to build an environment that looked organic but did not appear overly art-directed or require too many individual decisions,” Hutchens says. “We used procedural methods that were flexible and enabled happy accidents like those that occur in the natural world.”
Finished shots required the integration of live action elements, including actors and a partial set that were shot near Sydney. Getting those elements to blend with the background required detailed attention from RSP’s compositing team.
“We had to adjust the live plates, which were shot in the harsh Australian sun, to match the moody mix of light and shadow of our jungle,” explains compositing supervisor, Guido Wolter. “It required extreme care, especially at the edges of actors, to keep background lighting from pushing through, but the team accomplished it seamlessly. The results are fantastic.”
Extra effort was also required because the jungle environment had to be shared with other studios working on different shots. The entire expanse of the jungle needed to be fully detailed and viewable from any angle.
“It was a technical feat because every asset we created - from the library of plant species to cloud shaders - had to be easily accessible to other vendors using different software and methodologies,” notes Sarnes. “It required good communication, tight logistical coordination and a collaborative spirit.”
VFX producer Jeanne-Elise Prevost notes that the logistics of sharing huge assets between multiple studios were daunting.
“There was a lot to coordinate, particularly near the end when deadlines were approaching and shots had to be delivered,” she recalls. “Our approach was to stay positive and focused on the goal of getting the project done and ensuring everyone was happy with the work.”
In addition to creating the Ta Lo jungle environment, RSP contributed to several other aspects of the film. Notably, the studio collaborated with technologists from the Australian Institute for Machine Learning (AIML) to develop a novel technique for facial replacement using Artificial Intelligence. They used it in certain high-intensity combat scenes.
“We used machine learning tools to train neural network models of principal actors and apply them to the faces of stunt doubles engaged in martial arts fighting,” explains Wolter. “The output is incredibly real and sets a new bar for believability.”
While initially intended to be traditional 2D and 3D face replacements, the team opted for Artificial Intelligence, with Dr. John Bastian and Dr. Ben Ward building on the principals of Open Source “Deep Fake”.
Bastian explains the process by which “an 'identikit' of both the stunt double's and actor's faces is learned against a shared 'dictionary' of facial features. The faces are then detected in the plate, and the machine rebuilds the image using the identikit of the intended actor."
Ward adds, “We used approximately 30,000 face images across five characters, training five principal models that combined to over four million training iterations.”
The team’s work grew to a total of 51 face replacements over six key scenes. In more complex circumstances where training material was not available, performance was driven by digital doubles as the basis for the face replacement.
“This would have been unheard a couple of years ago,” notes Clark. “What would have been a costly, labor intensive process, is now in the hands of Machine Learning, with no special hardware. Where RSP differentiates itself is with the expertise of Dr. Bastian and Dr. Ward, and the collaboration with AIML, who worked closely with our creative team to produce solutions that demonstrate RSP’s is at the forefront of using AI in its pipeline.”
RSP’s previous work for Marvel includes Captain Marvel and Black Widow.