British rock band IDLES have unveiled a surprise collaboration, bringing together the video from Coldplay’s classic song ‘Yellow’ with IDLES’ new single ‘Grace’.
Commissioner John Moule approached production studio Joyrider with the project - posing the challenge of taking the renowned ‘Yellow’ music promo (which was released in 2000) and making it look like Chris Martin was singing IDLES’ new track.
View the MV here.
Research and development was performed by Joyrider’s director Jonathan Irwin, who utilised various AI tech solutions to land on the best way of making Chris Martin look like he was actually singing ‘Grace’, whilst maintaining the integrity of Chris’s original performance.
Initial tests on the mouth, face and full head replacement tests were trialed using AI tools such as wac2lip, which generated mouth shapes based on audio inputs in Stable Diffusion. However, this didn’t deliver the consistency and quality Joyrider required: Chris Martin sings front and center for the entire duration of the track without cutting away, so there could be no compromise on the quality or convincingness of the video.
The team then moved to an approach that used a filmed mouth singing the song, which the AI Deepfake training then used, before this pre-trained face went through around one million iterations to get to a very decent level.
The mouth had to be synced together seamlessly with the original video, augmenting Chris’ head shape and manipulating neutral faces in order to apply new mouth positions. To achieve this, Joyrider approached VFX specialists, Stone Dogs, to harness their flame compositing expertise. In a cyclic turn of events, Brian Carbin, senior VFX artist and co-founder of Stone Dogs who had worked on the original ‘Yellow’ video, was part of the team crafting this re-imagined iteration.
Wanting to make the video look more like Chris did over 20 years ago, the team were able to take the production to the next level and film the “real” Chris Martin in the band’s studio, singing the song at double speed (50fps) with three 4K cameras. The video was then played back and slowed down, adding an additional layer of subtlety to match the original plate. The filmed material was then Deepfaked further, helping to deliver a convincing AI performance of Chris Martin, filmed as if it actually happened in 2000.
Jonathan Irwin, director and AI technical lead, Joyrider, led weeks of further research and development and refining machine-learning approaches saying: “To create the training set, we fed every frame of the original video into DeepFaceLab and ran for five million iterations, approximately one and a half months of processing time, running 24/7. At each stage of improvement, there would be backwards and forwards of new AI training runs and flame VFX work, tracking, extensions, clean up, more tracking, test composites, more AI, more clean up etc.”
In edit, there was a further challenge of how to cut the original one shot ‘Yellow’ promo into a shorter IDLES track, without losing the opening sea at night setting or the bright walking away end scene.
Careful timing adjustments were needed to find the best fit for the IDLES lyrics to match Chris’ delivery as the footage is all in one take. A time/speed ramp effect was applied to a section of the film to bring the ‘Yellow’ and ‘Grace’ promos in sync. This speed effect was in keeping with the original creative narrative of time advancing. Additional time warp effects were positioned over certain points of the video, to take the viewer away from the original Coldplay ‘Yellow’ video and into an IDLES-esque world.
The grade was undertaken by Stone Dogs’ colourist Mark Meadows, who seamlessly blended in the new deep fake faces with the original footage, matching flesh tones and grain without affecting the original promo. The video begins with the beach and sky being somewhat dark until sunlight rises at around the video’s midpoint, therefore the new AI derived face had to match the changing colour grade.
CREDITS:
Production Company & AI Studio: Joyrider Video commissioner: John Moule
Creative direction and concept by Joe Talbot
Director: Jonathan Irwin
Producer/EP: Spencer Friend
AI Director & Technical Lead: Jonathan Irwin
Senior VFX Artists: Brian Carbin, Dave Kiddie, Rufus Blackwell Exec Post Producer: Danny Coster
Colourist: Mark Meadows
Data Wrangler: Jon Kerby
Post Production: Stone Dogs
1st AD: Bailey Marks
DP: Paul Mackay
Director & AI representative: Joyrider
Production Assistant: Karin Giat
1st AC: Daniel Kolditz
Video & Sound Playback: Von Adams Grip: Michael Farrell
Runner: Samad Olukine
Camera Equipment: Pete Moore at Focus Canning Special thanks to Focus Canning
Special thanks to directors James Frost & Alex Smith and the rest of the crew behind the original Yellow video, Coldplay and their management team.
Contains a sample from the music video for “Yellow” as performed by Coldplay, licensed courtesy of Parlophone Records Limited.
