As we continue to see the rise of creative machines that can automatically generate new synthetic content it’s likely that you’ve heard of an Artificial Intelligence (AI) derived technique known as “style transfer” – and if you haven’t heard of it then you’ve certainly seen it, whether you know it or not.
The process uses AI deep learning to apply the look and feel of one image to another, and appears in apps like Prisma and Facebook. These style transfers, however, are stylistic, and not photorealistic, and they look good because they look like they’ve been painted. Now a group of researchers from Cornell University and Adobe have augmented style transfer so that it can transfer the look of one photo onto another while still looking like a photo rather than just a painting, and the results are impressive.
The researchers’ work is outlined in a paper called “Deep Photo Style Transfer.” Essentially, they’ve taken the methods of the original style transfer, and added another layer of neural networks to the process — a layer that makes sure that the details of the original image are preserved.
“People are very forgiving when they see [style transfer images] in these painterly styles,” said Cornell professor Kavita Bala, a co-author of the study. “But with real photos there’s a stronger expectation of what we want it to look like, and that’s why it becomes an interesting challenge.”
The added neural network layer pays close attention to what Bala calls “local affine patches.” There’s no quick way to accurately translate this phrase, but it basically means the various edges within the image, whether that’s the border between a tree and a lake, or a building and the sky. While style transfer tends to play fast and loose with these edges, shifting them back and forth as it pleases, photo style transfer preserves them.
There are limits to the technique, of course. The algorithms seem to work best with structures like buildings, and the flaws are more obvious with faces. Furthermore, you can’t use massively different photos for transferring style, otherwise the neural networks have a tougher time analysing elements to copy from picture to picture.
“If you have a picture of a lake and you have a scene where you’re taking the style from, ideally it would also have a water body in it of some sort,” says Bala. “There’s no defined limit, but this is a good open research question. We put the code out because we want people to play with it and try it out.” (The code is available here on GitHub.)
The question is, how long will it be until we start seeing these sorts of photo style transfers being made accessible to the public, or before we see them being used in game and synthetic content development? My guess would be a couple of years. After all, the original style transfer went from a first research paper to Facebook’s app, reaching hundreds of millions of users, in less than two years’ time. And with Adobe’s involvement in this paper, there’s obviously an expectation that it’s at least a little bit interested in some sort of commercialisation.
For now, though, the researchers are already thinking about what areas photorealistic style transfer could be applied to next.
“The question of how far you can push it is important,” says Bala. “Video is a logical thing for it to go to, and that, I expect, will happen sooner rather than later, and of course game development could also be a fantastic opportunity where it would make a real difference to how fast new developers can create games.”
Matthew Griffin, described as “The Adviser behind the Advisers” and a “Young Kurzweil,” is the founder and CEO of the World Futures Forum and the 311 Institute, a global Futures and Deep Futures consultancy working between the dates of 2020 to 2070, and is an award winning futurist, and author of “Codex of the Future” series.
Regularly featured in the global media, including AP, BBC, Bloomberg, CNBC, Discovery, RT, Viacom, and WIRED, Matthew’s ability to identify, track, and explain the impacts of hundreds of revolutionary emerging technologies on global culture, industry and society, is unparalleled. Recognised for the past six years as one of the world’s foremost futurists, innovation and strategy experts Matthew is an international speaker who helps governments, investors, multi-nationals and regulators around the world envision, build and lead an inclusive, sustainable future.
A rare talent Matthew’s recent work includes mentoring Lunar XPrize teams, re-envisioning global education and training with the G20, and helping the world’s largest organisations envision and ideate the future of their products and services, industries, and countries.
Matthew's clients include three Prime Ministers and several governments, including the G7, Accenture, Aon, Bain & Co, BCG, Credit Suisse, Dell EMC, Dentons, Deloitte, E&Y, GEMS, Huawei, JPMorgan Chase, KPMG, Lego, McKinsey, PWC, Qualcomm, SAP, Samsung, Sopra Steria, T-Mobile, and many more.
FANATICALFUTURIST PODCAST! Hear about ALL the latest futures news and breakthroughs!SUBSCRIBE
1000's of articles about the exponential future, 1000's of pages of insights, 1000's of videos, and 100's of exponential technologies: Get The Email from 311, your no-nonsense briefing on all the biggest stories in exponential technology and science.