DARPA is giving its deep web search tech a huge upgrade

0 0

By Matthew Griffin Security and Privacy 13th December 2016

WHY THIS MATTERS IN BRIEF

The deep web is home to both legal and illegal institutions and activities that have always traditionally been hard to search, crawl and analyse but now a new upgrade from DARPA promises to change all that.

A lot of people have never heard of the US Governments “Memex” program. Memex is a deep web search engine that was first developed by a Stanford graduate working for Rescue Forensics on behalf of the Defense Advanced Research Projects Agency (DARPA) – the agency that created the original ARPANET, which then went on to form the foundation of the internet.

China's puts final touches to its 2,000km long unhackable quantum network

According to reports, because no one really knows for sure, the deep web is about 20 to a hundred times larger than the surface web – the part of the web that we all regularly search and trawl through and in order to access it, let alone search it, you need special tools. As a consequence Memex doesn’t work in the same way that Google or Bing do.

DARPA describes Memex as a set of search tools that government users can use to perform domain specific searches that display as rich, sophisticated infographics and apparently Memex can crawl over 95% of the deep web.

Memex’s original purpose in life was to try to help government agencies surface human trafficking, slavery and terrorist related content in an effort to help them try to stem the tides – in 2016 alone it’s thought that the deep web helped these organisations traffic over 21 million people and move over $500 Billion in funds, and as a result of its use it’s helped secure thousands of convictions.

South China Airlines swaps boarding cards for facial recognition

Rescue Forensics was recently acquired by IST Research, another company who also works with DARPA but this time on a platform called “Pulse”, a deep web search platform that works by looking at phone addresses and E-Mails as well as commonalities such as the spacing used in deep web adverts and aggregation sites, and that specialises in crawling deep web nets in parts of the world where security and accessibility aren’t great. Now the two are hooking up.

In the next year DARPA, IST Research and Research Forensics will join together and integrate Memex and Pulse to create a new, better platform and according to a report from Sophos the technology is becoming so good at what it does that if it keeps on being successful, we might need to rename what is currently the deep web.

Matthew Griffin / About Author

Matthew Griffin, described as “The Adviser behind the Advisers” and a “Young Kurzweil,” is the founder and CEO of the World Futures Forum and the 311 Institute, a global Futures and Deep Futures consultancy working across the next 50 years, and is an award winning futurist, and author of “Codex of the Future” series.

Regularly featured in the global media, including AP, BBC, Bloomberg, CNBC, Discovery, RT, Viacom, and WIRED, Matthew’s ability to identify, track, and explain the impacts of hundreds of revolutionary emerging technologies on global culture, industry and society, is unparalleled. Recognised for the past six years as one of the world’s foremost futurists, innovation and strategy experts Matthew is an international speaker who helps governments, investors, multi-nationals and regulators around the world envision, build and lead an inclusive, sustainable future.

A rare talent Matthew’s recent work includes mentoring Lunar XPrize teams, re-envisioning global education and training with the G20, and helping the world’s largest organisations envision and ideate the future of their products and services, industries, and countries.

Matthew's clients include three Prime Ministers and several governments, including the G7, Accenture, Aon, Bain & Co, BCG, Credit Suisse, Dell EMC, Dentons, Deloitte, E&Y, GEMS, Huawei, JPMorgan Chase, KPMG, Lego, McKinsey, PWC, Qualcomm, SAP, Samsung, Sopra Steria, T-Mobile, and many more.