Researchers say AI is advancing faster than our ability to understand it

WHY THIS MATTERS IN BRIEF

If capability keeps outrunning interpretability, society will deploy systems no one can fully explain or predict.

The capabilities of new Artificial Intelligence (AI) models are increasingly being discovered not invented according to researchers, and it is increasingly becoming more powerful, and mysterious.

Despite years of work on Explainable AI today’s most advanced systems remain black boxes for the most part. Scientists can observe what they do but cannot fully explain how they arrive at their conclusions or predict when they’ll fail. As Large Language Models (LLMs), the algorithmic engines behind popular chatbots, permeate society, researchers are warning that the window for understanding AI “minds” is rapidly closing even as the technology’s influence expands.

Last week, Eric Horvitz, chief scientific officer at Microsoft, and Robert West at EPFL in Switzerland of putting AI interpretability on the back burner. They call for new AI benchmarks and better tools for unpicking machine minds.

The challenge resembles efforts to understand our own minds. Some researchers have already taken a neuroscience-inspired approach, mapping AI’s internal networks to concepts, goals, and reasoning. Others borrow from psychology, treating AI as a participant of behavioural studies. And the stakes are rising.

AI tools already shape how people search for information, make decisions, and form judgments. Their answers influence everyday users and the researchers who build them.

As AI capabilities grow, our understanding of them could fall behind. “Preserving human agency must therefore remain a central goal,” the authors write.

LLMs are built on artificial neural networks (specifically, a design called the transformer). Inspired loosely by the brain, these networks connect vast numbers of artificial neurons into intricate architectures. The basic idea is straightforward. Data enters the network and passes through layers of computations, which transform it into an output like text or code.

At first, that output is often wrong. But with feedback and repeated training, the network adjusts the strengths of connections between neurons and gradually improves. It learns.

From AI to AGI: What Leaders Need to Know by Futurist Keynote Speaker Matthew Griffin

Book a Keynote with Matthew

After initial training, engineers turn to reinforcement learning, where algorithms improve through trial and error and further hone their responses. Another method, inspired by how the brain etches memories during sleep, reduces the tendency to forget old knowledge while learning new tasks. And self-attention, the key innovation behind transformers, allows AI to selectively focus on various words, images, sounds, or video frames at different moments, boosting efficiency and performance. Today, attention underpins nearly every major AI system.

Yet the inner workings of finished algorithms remain hidden.

to AI’s black box examined how artificial neurons responded to images, revealing that neural networks build increasingly more sophisticated “ideas” of the world. Google Brain borrowed methods from cognitive psychology to study AI behaviour, while others investigated whether LLMs could mimic aspects of “theory of mind” – the ability to infer what others are thinking and feeling.

These studies laid the foundation for a popular method called mechanistic interpretability. , creator of Claude, is leading the field. Company researchers have linked patterns of algorithmic activity to specific concepts and reverse engineered parts of neural networks to expose how internal computations shape responses.

Other tech giants are joining the cause. OpenAI that work in more explainable steps and building that pause, “think,” and justify their conclusions in plain language. is building microscope-like tools for neural networks, helping researchers peer into their decision-making process. And Microsoft has released new aimed at responsible use of AI.

Understanding AI, the authors write, does not require tracing every line of code or every neural-network parameter. Just as neuroscience, psychology, and sociology offer different windows into human behaviour, AI can be studied at multiple levels, from how individual circuits work to observing behaviour in real-world scenarios.

The challenge is that AI capabilities may be advancing faster than our ability to explain them. And some researchers believe time is running out.

Three trends are making AI more opaque.

The first is how we evaluate AI. Increasingly, LLMs we being used to train, benchmark, and improve other models. AI now score metrics like helpfulness, rank competing outputs, detect hallucinations, and assess new releases. In a system known as constitutional AI, for example, algorithms critique their own responses using reinforcement learning and generate explanations for their reasoning. researchers have proposed AI debate frameworks, where multiple models challenge each another’s conclusions before a human has the last say. Researchers are also exploring automated interpretability tools. Like digital neuroscientists, AI systems are used to analyse each other – describing neurons, circuits, and behavioural patterns – to explain increasingly complex models.

Using AI to solve an AI-induced problem introduces a paradox. If AI-generated explanations become too complex for humans to verify, opacity compounds.

A second trend is the rise of AI societies like the OpenClaw social network I’ve spoken about before. Networks of interacting AI agents are becoming more common, particularly in complex tasks such as scientific research and drug discovery. Yet as they become more sophisticated, their communication could drift from human language – such as the time we saw Google’s AI develop its own language – and reasoning, making them harder to interpret.

Studying their interactions with methods adapted from sociology could unveil unexpected norms, hidden rules, and collective behaviour. The authors argue that training in the future should not only reward effective collaboration among AI agents, but also ensure humans can understand their communication.

The last trend already permeates our lives. ChatGPT, Claude, Gemini, and other LLMs listen to our woes, offer recipes, and code websites. But they also learn about humanity. Through training data and interactions, they glimpse how people think, reason, and feel. In turn, they capture core aspects of life, such as fear, anxiety, happiness, and the need for social belonging.

To be clear, the systems don’t have intentions. They’re not examining us. But even as we struggle to understand them, AI systems are building more sophisticated models of who we are.

“A striking asymmetry follows: While human understanding of AI declines, AI understanding of humans deepens, producing new forms of behavioural opacity,” the authors write.

But complacency is perhaps even more insidious. AI assistants are often optimised to be agreeable, helpful, sycophantic and reassuring. Studies have found that people generally prefer AI agents that support their opinions and decisions. As AI is woven into everyday life, curiosity and scepticism may gradually give way to trust. They work. Why question how?

The authors don’t have a solution for the long-standing problem. Instead, they call for better benchmarks to measure AI capabilities and stronger evaluation methods. And while open-source projects and crosstalk between commercial companies and academia are now frequent, they say we need lasting norms of responsible disclosure. Mechanistic interpretability and AI “psychology” could build on each other.

“The goal is not just more capable AI, but AI that is more intelligible, accountable, and aligned with human aims,” they write.

Why are researchers worried about understanding modern AI?
Today’s most advanced models remain black boxes — observable but not explainable — and interpretability work is lagging as the technology’s reach rapidly expands.