Nvidia’s new Vera Rubin racks eliminate cables and fans

Last week Nvidia held a media event that included a field trip to an undisclosed location in Silicon Valley - a building without any identifying signage. Though hundreds of racked GPUs were chugging away at real Artificial Intelligence (AI) tasks inside, Nvidia calls the facility an “engineering superlab” rather than a data centre. Among the AI being crunched was a test version of OpenAI’s GPT model optimised for Nvidia’s new Vera Rubin platform, named after the legendary astronomer.

For show-and-tell purposes, several powerful computers were on display in the building’s lobby. Containing GPUs, CPUs, and supporting components and known as trays, they slide into racks like the ones in the superlab. Data centers equipped with vastly larger quantities of these racks power a huge percentage of the AI in our lives.

Basically, we all use these Nvidia tray computers all day long. It’s just that under normal circumstances, we never see them or even have much reason to give them any thought. For the reporters attending getting a close-up peek at the ones at Nvidia’s unmarked superlab was a rare opportunity to contemplate how they do what they do.

For the sake of comparison, the ones in the lobby included a Vera Rubin tray sitting next to one based on Nvidia’s previous-generation platform for its GB200 and GB300 chips. At a glance, it was obvious that the redesign went far beyond the new tray’s more powerful chips. The older tray looked like a rat’s nest, with an array of cables snaking every which way to connect components. The Vera Rubin tray was radically more streamlined. It was cable free. Well, not literally cable free per se.

“We call it cableless - it actually has two cables,” explained Senior VP of Hardware Engineering Andrew Bell, a 25-year Nvidia veteran. One cable remains to hook up the tray to power, another to connect it to networking resources. But they were unobtrusive enough that you don’t notice them until they get pointed out.

It also might be unfair to call the earlier design a rat’s nest, but when a rack has 50 or 60 cables, as they normally do, they give off an overwhelming impression of chaos. Especially when it’s sitting next to one that’s so close to being cable-free.

The Future of AI, Cyber, and Data 2030, a keynote by futurist Matthew Griffin

Two overarching questions are relevant to this redesign: Why did Nvidia decide to go cableless, and how did the company pull it off? The first answer might be pretty obvious if you’ve ever built a desktop PC, or at least opened one up.

Inside a computer, cables just complicate things. Some 50 or 60 cables add up to 100 to 120 additional steps during assembly. Each connection is a place something could go wrong during the building process and, after that, a potential point of failure. Cables create thickets that get in the way of other components, both during manufacturing and later servicing.

“It takes two to three hours to assemble ,” Bell said. “And when we first started building it, you had about a 20% chance of getting it right the first time.”

Ian Buck, VP and GM of hyperscale and high-performance computing, added, “Just imagine yourself plugging in every one of those cables and trying to do it at a scale of Nvidia.”

The success rate did gradually increase, but it wasn’t just Nvidia’s own scale that was an issue. Rather than mass-producing trays itself, the company shipped parts to contractors and manufacturing partners, such as Dell. Each time production expanded, it set off a Groundhog Day-style cycle as new teams learned how to assemble the cable-heavy trays.

If cables’ downsides are so many and varied, why have they been so fundamental to computer designs for so long? One way or another, components do need to talk to each other. For a long time, it wasn’t obvious how to achieve that sans cabling – even though today we’re starting to see Terahertz RF technologies emerge with the aim to also make data centres cable free.

For Nvidia, the breakthrough cascaded out of another major design achievement reflected in the Vera Rubin platform: It’s fanless. Instead of using air cooling to prevent overheating, the tray is 100% liquid cooled, using a new approach that works with water at up to 113 degrees Fahrenheit, dramatically reducing energy requirements compared to earlier liquid-cooling techniques.

As a bonus, removing fans also promises to dramatically reduce the noise pollution inside data centres. At the superlab, which is full of fan-cooled racks everyone wore earplugs, and Bell spoke with a microphone.

When Nvidia figured out how to ditch the fans, it opened up room inside the Vera Rubin trays. More room allowed the company to create a midplane connector, based on its NVLink technology, that established cable-free communications between components. Even if Nvidia had somehow found room for the midplane connector in an air-cooled tray, it wouldn’t have worked: It would have obstructed airflow.

In short, one elegant technical breakthrough enabled another. You don’t need to get that backstory from Nvidia executives to see the elegance in the resulting Vera Rubin tray. It’s sleek, black, and bordering on brutalism in its lack of ornamentation. Compared to it, its predecessors look like highly compressed versions of 1951’s Univac.

All of this is ultimately in service of a faster, more reliable ramp-up of the Vera Rubin platform, in factories that can now be more fully automated using humanoid robots to build servers and assemble racks.

“This tray can be assembled in five minutes instead of two to three hours,” Bell said. “The first time you put it together, you have a 95% chance of getting it right. You can take it to a new factory. The first time they put it together, it’s 95%. We expect after a month, they’ll be at 98% or 99%.”

With other AI chip makers ever more intent on stealing market share from Nvidia those efficiencies matter. But even if Nvidia wasn’t after elegance for elegance’s sake, the company has achieved it.