British rock band Idles recently released a music video for their “Grace” track that puts an AI twist on Coldplay’s “Yellow” music video from 2000. The project called on production studio Joyrider (www.joyriderfilms.com) and VFX experts Stone Dogs (https://stonedogs.co.uk) to alter the original video, which was directed by James Frost and Alex Smith. In the 2000 release, Coldplay’s Chris Martin sings while slowly walking along a dark and foggy beachfront as the sun begins to rise. Idles’ version of the visual has been retimed to coincide with the tempo of “Grace,” and Martin’s appears to be singing the different lyrics for the song.
Research and development was performed by Joyrider’s director Jonathan Irwin, who used various AI solutions to land the best way of making Martin look like he was actually singing “Grace” while maintaining the integrity of his original performance. Initially, the AI tool Wac2lip, which generates mouth shapes based on audio inputs in Stable Diffusion, was used to make it look like Martin was signing the new lyrics, but consistency and quality proved challenging, as he’s front and center for the entire duration of the video. The team then moved to an approach that used a filmed mouth singing the song, which the AI deepfake training then used. The mouth had to be synced together seamlessly with the original video, augmenting Martin’s head shape and manipulating neutral faces in order to apply new mouth positions. To achieve this, Joyrider approached VFX specialists Stone Dogs to harness their Flame compositing expertise. Coincidentally, Brian Carbin, senior VFX artist and co-founder of Stone Dogs, was part of the team that worked on the original Yellow video.
Wanting to make the video look more like Martin did over 20 years ago, the team was able to take the production to the next level and film the Martin at the band’s studio, where he sang the Grace lyrics at double speed (50fps) with three 4K cameras. The video was then played back and slowed down, adding an additional layer of subtlety to match the original plate. The filmed material was then deepfaked further, helping to deliver a convincing AI performance of Martin, filmed as if it actually happened in 2000.
The VFX team included Brian Carbin, Dave Kiddie, Rufus Blackwell and Danny Coster. Irwin, who served as director and AI technical lead for Joyrider, led weeks of further R&D.
“To create the training set, we fed every frame of the original video into DeepFaceLab and ran for five million iterations, approximately one-and-a-half months of processing time, running 24/7,” Irwin explains. “At each stage of improvement, there would be backwards and forwards of new AI training runs and Flame VFX work, tracking, extensions, clean up, more tracking, test composites, more AI, more clean up, etc.”
In edit, there was a further challenge of how to cut the original Yellow piece into a shorter Idles track without losing the opening sea-at-night setting or the bright, walking-away end scene. Careful timing adjustments were needed to find the best fit for the Idles lyrics to match Martin’s delivery, as the footage is all in one take.
A time/speed ramp effect was applied to a section of the film to bring the Yellow and Grace videos in sync. This speed effect was in keeping with the original creative narrative of time advancing. Additional time warp effects were positioned over certain points of the video to take the viewer away from the original Yellow video and into an Idles-esque world.
The grade was undertaken by Stone Dogs’ colorist Mark Meadows, who seamlessly blended in the new deep fake faces with the original footage, matching flesh tones and grain without affecting the original look.