Pokémon Go changed our children’s lives – okay, we’ll admit it – all of our lives. Now it’s hard to imagine life without those plucky virtual creatures sneaking up on you when you’re trying to have some alone time.


But get ready for it, because researchers from MIT have kicked the augmented reality game up a notch by inventing a program that allows virtual objects like Pokémon to interact in real time with real world environments. Yep, this means one day you could be snaring a Ponyta as it prances through a field or wasting 20 Pokéballs on a Zubat that keeps messing up your curtains.


K-Pop's newest band member will be a CGI generated digital human


The technology, dubbed Interactive Dynamic Video (IDV) not only allows animated characters move about the world but it also allows them to realistically – and this is the cool bit, affect objects in the environment, like Pikachu rustling the leaves of a bush or pushing a swing.

“IDV technique lets us capture the physical behaviour of objects, which gives us a way to play with them in augmented reality,” said Abe Davis from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).

“By making videos interactive, we can predict how objects and characters will respond to unknown forces and explore new ways to engage with videos.”


How it works

Show me the magic

The technology uses traditional cameras to analyse the tiny, almost invisible vibrations given off by the objects in the frame. These vibration patterns are then fed through a number of algorithms to predict how that object can move. Based on data captured from just 5 seconds of footage, the program can build realistic simulations of an object’s motions right there on the screen in front of you.


Scientists ran an experiment to prove a super intelligent AI couldn't be controlled


“If you want to model how an object behaves and responds to different forces, we show that you can observe the object respond to existing forces and assume that it will respond in a consistent way to new ones,” says Davis, adding that they’ve even made it work on some existing YouTube videos.

Not only does the technique allow programmers to drop Pokémon or other animated characters into an environment and have them realistically interact with it, it also works in reverse, allowing users to interact with a virtual world, and poke and prod at objects ‘inside’ video footage. Think plucking at the strings of a guitar in a video, or telekinetically controlling the leaves of a bush.

“When you look at VR companies like Oculus, they are often simulating virtual objects in real spaces,” says Davis. “This sort of work turns that on its head, allowing us to see how far we can go in terms of capturing and manipulating real objects in virtual space.”

Their research has yet to appear in a peer reviewed journal, but you can read their recent report online here, and check out more of its capabilities in the video above. We’re so ready to have this in our life and we know you are too.

About author

Matthew Griffin

Matthew Griffin, described as “The Adviser behind the Advisers” and a “Young Kurzweil,” is the founder and CEO of the World Futures Forum and the 311 Institute, a global Futures and Deep Futures consultancy working between the dates of 2020 to 2070, and is an award winning futurist, and author of “Codex of the Future” series. Regularly featured in the global media, including AP, BBC, Bloomberg, CNBC, Discovery, RT, Viacom, and WIRED, Matthew’s ability to identify, track, and explain the impacts of hundreds of revolutionary emerging technologies on global culture, industry and society, is unparalleled. Recognised for the past six years as one of the world’s foremost futurists, innovation and strategy experts Matthew is an international speaker who helps governments, investors, multi-nationals and regulators around the world envision, build and lead an inclusive, sustainable future. A rare talent Matthew’s recent work includes mentoring Lunar XPrize teams, re-envisioning global education and training with the G20, and helping the world’s largest organisations envision and ideate the future of their products and services, industries, and countries. Matthew's clients include three Prime Ministers and several governments, including the G7, Accenture, Aon, Bain & Co, BCG, Credit Suisse, Dell EMC, Dentons, Deloitte, E&Y, GEMS, Huawei, JPMorgan Chase, KPMG, Lego, McKinsey, PWC, Qualcomm, SAP, Samsung, Sopra Steria, T-Mobile, and many more.

Your email address will not be published. Required fields are marked *