Scroll Top

Research of billions of links shows the internet is both shrinking and expanding



The internet is increasingly our second, if not our first, home – and changes to it affect all of us in one way or another.


Love the Exponential Future? Join our XPotential Community, future proof yourself with courses from XPotential Universityconnect, watch a keynote, or browse my blog.

The online world is continuously expanding – there’s always more services, more users, and more activity. And last year the number of websites registered on the “.com” domain surpassed 150,000,000.

However, more than a quarter of a century since its first commercial use, the growth of the online world is now slowing down in some key categories. And a team in Australia, who have just finished a multi-year research project to analyze global trends in “online diversity and dominance” have just published their results.


GE's revolutionary industrial machines can have conversations with engineers


Their research, published the other week in Public Library of Science, is the first to reveal some long term trends in how businesses compete in the age of the web, and on the overall state of web based competition and innovation itself.

“We saw a dramatic consolidation of attention towards a shrinking, but increasingly dominant, group of online organisations,” said the researchers, adding, “So, while there is still growth in the functions, features, and applications offered on the web, the number of entities providing these functions is shrinking.”

In all they analysed more than six billion user comments from the social media website Reddit dating back to 2006, as well as 11.8 billion Twitter posts from as far back as 2011. And in total, their research used a massive 5.6Tb trove of data from more than a decade of global activity.


Space based communication company Starlink just signed its first airline customer


With the Reddit posts they analysed all the links to other sites and online services – more than one billion in total – to understand the dynamics of link growth, dominance, and diversity through the decade.

They also used a measure of link “uniqueness.” On this scale, 1 represents maximum diversity, in other words all links have their own domain, and 0 is minimum diversity – all the links are on one domain, such as “”.

A decade ago, there was a much greater variety of domains within links posted by users of Reddit, with more than 20 different domains for every 100 random links users posted. Now there are only about five different domains for every 100 links posted.


This astronaut riding a horse shows AI is getting better at creating synthetic content


In fact, between 60 to 70 percent of all attention on key social media platforms is focused towards just ten popular domains.

Beyond social media platforms, they also studied linkage patterns across the web looking at almost 20 billion links over three years. These results reinforced the “rich are getting richer” online.

The authority, influence, and visibility of the top 1,000 global websites, as measured by network centrality or PageRank, is growing every month, at the expense of all other sites.

The web started as a source of innovation, new ideas, and inspiration – a technology that opened up the playing field. It’s now also becoming a medium that actually stifles competition and promotes monopolies and the dominance of a few players.


Check Point says Russian hackers are trying to break into ChatGPT to get up to no good


Their findings resolve a long running paradox about the nature of the web – does it help grow businesses, jobs, and investment? Or does it make it harder to get ahead by letting anyone and everyone join the game? The answer, it turns out, is it does both.

While the diversity of sources is in decline, there is a countervailing force of continually increasing functionality with new services, products, and applications – such as music streaming services like Spotify, file sharing programs like Dropbox, and messaging platforms like Messenger, Whatsapp, and Snapchat.

Another major finding was the dramatic increase in the “infant mortality” rate of websites  – with the big kids on the block guarding their turf more staunchly than ever.


Miami becomes the first city to mine crypto and give citizens dividends


The researchers examined new domains that were continually referenced or linked-to in social media after their first appearance, and found that while almost 40 percent of the domains created 2006 were active 5 years on, only a little more than 3 percent of those created in 2015 remain active today.

The dynamics of online competition are becoming clearer and clearer. And the loss of diversity is naturally concerning. Unlike the natural world, there are no sanctuaries, competition is part of both nature and business.

The study has profound implications for business leaders, investors, and governments everywhere. It shows the network effects of the web don’t just apply to online businesses. They have permeated the entire economy and are rewriting many previously accepted rules of economics.

For example, the idea that businesses can maintain a competitive advantage based on where they are physically located is increasingly tenuous. Meanwhile, there’s new opportunities for companies to set up shop from anywhere in the world and serve a global customer base that’s both mainstream and niche.


Google's AI can rate photos on aesthetic appeal


The best way to encourage diversity is to have more global online businesses focused on providing diverse services, by addressing consumers’ increasingly niche needs.

In Australia, for example, this is starting to emerge via homegrown companies such as CanvaSafetyCulture, and iWonder. But, hopefully many more will appear in the decade ahead that will help reverse the current trends.

Related Posts

Leave a comment


1000's of articles about the exponential future, 1000's of pages of insights, 1000's of videos, and 100's of exponential technologies: Get The Email from 311, your no-nonsense briefing on all the biggest stories in exponential technology and science.

You have Successfully Subscribed!

Pin It on Pinterest

Share This