Decades after CGI appeared rotoscoping is still one of the most laborious special effects tasks, but AI’s now tackling the problem.


Interested in the Exponential Future? Connect, download a free E-Book, watch a keynote, or browse my blog.

Artificial Intelligence (AI) is being used to create more and more synthetic content, in the form of audio, text, video, and now synthetic special effects, and back in 2018, inside an old auto body shop in Silicon Valley a gentleman called Stefan Avalos pushed a movie camera down a dolly track. He and a small crew were making a short film about self-driving cars. They were shooting a rusted blue 1963 Austin Mini that, via special effects, would be transformed into an autonomous vehicle that looked more like the DeLorean from the movie “Back to the Future.”


Invidia's Canvas puts a professional AI artist in the palm of your hand


Stepping back from the camera at the time Avalos referred wryly to the movie he was filming as “Project Unemployment” because the film he and his crew were making was a way of testing new technology from a start-up called Arraiy, a company that at the time was trying to automate the creation of digital effects for movies, television and games, that then went on to be bought by Matterport – all tasks that today are as tedious as they are laborious – using a new AI system.


Under the hood of “The Human Race” by The Mill

The system in question was also being developed by software giants Adobe and AutoDesk, as well as in other technology industry research labs, including Nvidia, and ultimately, the hope at the time, which has been born out since, was that once the technology was good enough and embedded into studio workflows it would spell the end to all the drudgery associated with creating and refining special effects – as well as many of the people involved in the process. As is the way with automation.

Since the advent of Jurassic Park back in the 80’s, Computer Generated Imagery (CGI) has transformed how movies and TV series are made. But for all CGI’s promise creating digital effects is still a painstaking and enormously tedious process. For every second of movie time armies of designers can spend hours isolating people and objects in raw camera footage, digitally building new images from scratch and combining the two as seamlessly as possible to create the final result that you see on the big screen.


Scientists built the largest AI supercomputer yet to create brain scale AI


Arraiy, the company behind the system Alvaros was testing, was one of the first to use AI to automate as many of these laborious tasks as possible, and thanks to improvements in deep learning we’re now at the point where these systems can edit noise and mistakes out of images as well as, as you’ve seen elsewhere in this codex, create a wide range of synthetic content.

Inside Arraiy’s offices teams of data scientists were hard at work building computer algorithms that could learn design tasks by analysing years worth of work by movie effects houses. That includes systems that learn to rotoscope raw camera footage – carefully separating people and objects from their backgrounds so that they can be dropped onto new backgrounds.

In order to train their algorithms the team collected a decade of rotoscoping and other visual effects work from various design houses, as well as adding their own work to the collection. After filming people, mannequins and other objects in front of a classic green screen company engineers were quickly rotoscope thousands of images relatively quickly that could then be added to the training corpus, and once the algorithm was trained the team unleashed it and let it rotoscope images without help from a green screen.


General Motors starts is using AI to design its future cars


This is the work that Avalos tested while all the while thinking the work could ultimately supplant work done by his own effects house. But as he saw it one of the big upsides in his case was the fact he would no longer have to outsource the tedious work the system was helping automate to workers in other countries.

Meanwhile, elsewhere, Kognat debuted its Rotobot deep learning rotoscope and compositing tool that it made available with NUKE that, similarly, thanks to its intense training is able to isolate all of the pixels that belong to a certain class into a single mask, called segmentation. The effect is the isolation of portions of the image, just like rotoscoping that can isolate all of the pixels of a single instance of a class, for example where a class could be a “person” or a “cat,” and so on, into its own layer.


New blockchain DNS system would put an end to DDoS attacks


As hardware improves these techniques and tools will soon be able to help Special effects artists generate preliminary effects right on the movie set in real time, and help give filmmakers a better idea of what the finished product will look like – a challenge that was then notably taken up by The Mill, a special effects house based in London, that a year before had taken the same system and used it to create one that could instantly apply effects that changed the look of cars in their critically acclaimed film “The Human Race” that you can watch above.

About author

Matthew Griffin

Matthew Griffin, described as “The Adviser behind the Advisers” and a “Young Kurzweil,” is the founder and CEO of the World Futures Forum and the 311 Institute, a global Futures and Deep Futures consultancy working between the dates of 2020 to 2070, and is an award winning futurist, and author of “Codex of the Future” series. Regularly featured in the global media, including AP, BBC, Bloomberg, CNBC, Discovery, RT, Viacom, and WIRED, Matthew’s ability to identify, track, and explain the impacts of hundreds of revolutionary emerging technologies on global culture, industry and society, is unparalleled. Recognised for the past six years as one of the world’s foremost futurists, innovation and strategy experts Matthew is an international speaker who helps governments, investors, multi-nationals and regulators around the world envision, build and lead an inclusive, sustainable future. A rare talent Matthew’s recent work includes mentoring Lunar XPrize teams, re-envisioning global education and training with the G20, and helping the world’s largest organisations envision and ideate the future of their products and services, industries, and countries. Matthew's clients include three Prime Ministers and several governments, including the G7, Accenture, Aon, Bain & Co, BCG, Credit Suisse, Dell EMC, Dentons, Deloitte, E&Y, GEMS, Huawei, JPMorgan Chase, KPMG, Lego, McKinsey, PWC, Qualcomm, SAP, Samsung, Sopra Steria, T-Mobile, and many more.

Your email address will not be published. Required fields are marked *