27 views
0

WHY THIS MATTERS IN BRIEF

Not only does this technology have benefits for video search, but it also has surveillance and privacy implications.

 

Recently I talked about a new Artificial Intelligence (AI) advancement from Hitachi that let’s its AI’s pick out and follow individuals in a crowd – something that would be especially useful in China, or any surveillance state, if they wanted to track and monitor someone as they moved around. And now that technology has a natural accomplice. The bleeding edge of computer science these days is all about making computers more like humans. We’re using neural networks to help machines recognize objects, play games, and even speak in a more realistic way. In a new feat of machine learning magic, Google Research has developed a system that can replicate the “cocktail party effect,” where your brain focuses on a single audio source in a crowded room. The results are impressive — almost worryingly so.

 

RELATED
Robot surgeon STAR bests its human counterparts in world first

 

Google calls this technique “Looking to Listen” because it watches videos with multiple speakers to split up the audio – it uses both auditory and visual signals, just like your brain does. There’s nothing special about these videos, either. They’re just videos with a single audio track consisting of more than one person.

 

The tech in action
 

To build a tool capable of this, Google started with 100,000 samples of high quality lectures and talks from YouTube. Engineers chopped up the videos to get segments of clean speech with clearly visible speakers and no background noise. That left Google Research with 2,000 hours of video consisting of a single person speaking, they call this the AVSpeech data set. The trick was using these clean samples to create “fake” cocktail parties. The researchers combined the videos, so multiple people were speaking. That’s the data Google used to train its neural network.

Like many other Google Research breakthroughs, this one used a convolutional neural network. The input to the network consists of visual features of the speakers as well as the spectrogram of the video’s soundtrack. By processing the video, the network learns how to separate the video into a “time frequency mask” for each speaker. The output mask is matched up with the audio input spectrogram to generate separate audio tracks.

 

RELATED
South Korea introduces the world's first "Robot Tax"

 

With the training done, Google unleashed the network on new videos. As you can see in Google’s examples, this works surprisingly well. The Looking to Listen model can identify what audio is coming from a speaker, and filter out everything else. This technology could have applications in video conferencing, hearing aids, and video surveillance.

On that last point, this technology could be so powerful that it’s not hard to imagine scenarios where it’s abused. With future speed and accuracy improvements, an observer could pick out your voice on a crowded street to find out what you said, and while there’s no indication Google has any intention of doing that they aren’t alone in doing neural network research for these purposes.

About author

Matthew Griffin

Matthew Griffin, award winning Futurist working between the dates of 2020 and 2070, is described as “The Adviser behind the Advisers” and a “Young Kurzweil.” Regularly featured in the global press, including BBC, CNBC, Discovery and RT, Matthew’s ability to identify, track, and explain the impacts of hundreds of revolutionary emerging technologies on global culture, industry and society, is unparalleled. Recognised for the past six years as one of the world’s foremost futurists, innovation and strategy experts Matthew is an international speaker who helps governments, investors, multi-nationals and regulators around the world envision, build and lead an inclusive, sustainable future. A rare talent Matthew sits on several boards and his recent work includes mentoring Lunar XPrize teams, building the first generation of biological computers and re-envisioning global education with the G20, and helping the world’s largest manufacturers ideate the next 20 years of intelligent devices and machines. Matthew's clients include three Prime Ministers and several governments, including the G7, Accenture, Bain & Co, BCG, BOA, Blackrock, Bentley, Credit Suisse, Dell EMC, Dentons, Deloitte, Du Pont, E&Y, HPE, Huawei, JPMorgan Chase, KPMG, McKinsey, PWC, Qualcomm, SAP, Samsung, Sopra Steria, UBS, and many more.

Your email address will not be published. Required fields are marked *