Groq's ultrafast LPU accelerator smashes AI LLM benchmarks

By Matthew Griffin Computing 20th August 2024

WHY THIS MATTERS

When it comes to AI in some cases fast is best and a new rival GPU manufacturer has just one upped Nvidia.

Love the Exponential Future? Join our XPotential Community, future proof yourself with courses from XPotential University, read about exponential tech and trends, connect, watch a keynote, or browse my blog.

Groq, led by ex-Google engineer and CEO Jonathan Ross, claims to have created the first ever Language Processing Unit (LPU) which it says can deliver the fastest speeds for AI applications. It’s a bold claim, but one that the latest demos more than back up, suggesting it could well become an absolute game-changer for AI.

Researchers turn plants into biological computers to grow better crops

Ross, who previously designed Google’s Tensor Processing Unit (TPU), launched Groq in 2016 to create a chip capable of executing deep learning inference tasks more efficiently than existing CPUs and GPUs.

The company’s Tensor Stream Processor (TSP) is likened to an assembly line, processing data tasks in a sequential, organized manner. In contrast, a GPU is akin to a static workstation, where workers come and go to apply processing steps. The TSP’s efficiency became evident with the rise of Generative AI, leading Ross to rebrand the TSP as the Language Processing Unit (LPU) to increase its recognizability.

See it in action.

Unlike GPUs, LPUs utilize a streamlined approach, eliminating the need for complex scheduling hardware, ensuring consistent latency and throughput. LPUs are also energy efficient, reducing the overhead of managing multiple threads and avoiding underutilization of cores. Groq’s scalable chip design allows multiple TSPs to be linked without traditional bottlenecks, simplifying hardware requirements for large-scale AI models.

For the first time a flying yacht has made green hydrogen at sea

The first public demo of Groq was a lightning-fast AI answers engine that generated answers with hundreds of words in less that a second. Matt Shumer posted the test on X and says more than 3/4 of the time was spent searching not generating.

If you want to try Groq for yourself, to get an idea of just how fast it can be for AI, go to this chat page. Use the drop down on the left to switch between the different available models.

Matthew Griffin / About Author

Described by NASA as a "Walking Encyclopaedia of the Future" and one of the world’s most in demand keynote speakers Matthew Griffin, Founder and Futurist in Chief of the 311 Institute, a global Futures and Deep Futures advisory firm, is a multi-award winning Futurist and strategic foresight expert, author, lecturer, mentor, podcast and YouTube host.

With a distinguished career running global business units at Atos, EMC, and IBM today Matthew helps leaders and titans of industry from all over the world explore the impact of the inter sections of technology, culture, and leadership, then helps them create high performance organisations that can continuously disrupt, outpace, and outthink their competition.

15 times author of the hit series the Codex of the Future series Matthew's ability to identify, track, and explain the impacts of hundreds of emerging technologies and trends on global business, culture, and society has earned him a powerful reputation and a roster of clients that include royal households, world leaders across the G7, G20, and G77, and many of the world's most recognised and exceptional organisations including ABB, Accenture, ABN Amro, Adidas, AON, ARM, BCG, Centrica, Citi Group, Coca Cola, Deloitte, Dentons, Disney, Dow, EY, Google, KPMG, Lego, Microsoft, Legal & General, LinkedIn, Nokia, Paramount, PepsiCo, PWC, Qualcomm, RWE, Samsung, T-Mobile, UBS, Visa, WIRED, WPP and hundreds of others.

Matthew is regularly featured in the global media including the AP, BBC, Bloomberg, CNBC, Discovery, Forbes, Khaleej Times, Telegraph, TIME, ViacomCBS, WIRED, and the WSJ, and his mission is to help organisations create a fair and sustainable future whose benefits are shared by everyone irrespective of their ability, background, or circumstances.