Data center experts on energy use for AI: 'Calm the heck down'

There’s no shortage of anxiety about the energy demands of artificial intelligence.

In mid-January, OpenAI CEO Sam Altman told a crowd at Davos that breakthroughs in nuclear fusion will be essential to powering AI’s rising appetite for energy. (Altman is personally invested in fusion startup Helion and backed IPO plans of the microreactor startup Oklo.) A week later, the International Energy Agency predicted that global data center energy use could double by 2026, largely thanks to AI and crypto mining.

But data center experts have mixed views on the scope of the problem. And while data center operators are rushing to meet a sudden surge of computing needs from AI, there’s optimism that advancements in software, AI models, grid infrastructure, and co-location strategies with renewables can help.

“It’s not like we’re helpless in the face of this massive increase in the demand for AI,” said data center and energy researcher Jonathan Koomey. “There’s all sorts of things we can do, and we’re only at the beginning of doing those things.”

Research often overestimates the electricity use and carbon footprint of technologies, Koomey said, in part because they underestimate energy efficiency gains. 

In the 1990s, some researchers believed the internet would consume half of the country’s electricity in a decade. But innovations in server design, cooling techniques, and software to manage computational loads kept data center energy use at between 1% and 1.3% of global demand in 2022. Meanwhile, global internet traffic has grown 25-fold since 2010, according to IEA.

“We don’t know how much electricity AI will use next year, never mind five years from now, and anyone who comes in with a projection beyond that is just nuts, because [information and communication tech] changes so fast,” Koomey said.

Koomey pointed to a recent report by Schneider Electric that contains more conservative projections. By Schneider’s estimates, total data center energy demand could reach around 800 TWh by 2028, compared to the IEA projection of 1,000 TWh by 2026. AI workloads will grow more rapidly, the report found, accounting for up to 20% of total data center demand by 2028. (The Schneider report doesn’t mention loads from crypto, however, so the two are not perfectly comparable.) 

Of course, given the many uncertainties, there’s a good chance Schneider’s numbers aren’t quite right either, Koomey added: “But the point is, until someone does a careful drill down into that IEA study and into the other data, we don’t really know.”

Tech tools

There’s no doubt that data center electricity needs are increasing, and fast. And while the industry has so far relied on better, smaller transistors for performance and efficiency improvements in servers, it’s generally agreed that the end of miniaturization is near. That makes operational improvements much more critical.

“We’re running out of tricks on the hardware side,” Koomey said, adding that companies need to stop relying on the “manna from heaven that came from shrinking transistors” and start pushing design changes instead.

A lot of these optimization efforts are already underway. In particular, said Boris Gamazaychikov, a senior manager on the emissions reduction team at Salesforce, there is a push to utilize more energy efficient AI models.

“It’s really important, when we look at AI, to ask ourselves if the most general purpose, largest models are necessary,” Gamazaychikov said. The efficiency of a given model is directly related to how many parameters it has, he added, and people are beginning to realize they can rely on more fine-tuned and domain-specific models. Today, some of the largest models (trained on the entire corpus of the internet) have a trillion parameters, but there are high-functioning models with as few as a billion parameters.

And orchestration can also reduce a model’s energy use, he added: “Instead of having this giant model that’s super inefficient, trying to parse through the entire internet, you have these smart, smaller, modular models that are more efficiently getting that information.”

That’s a trend that extends to computing giants as well. Google reports that selecting efficient machine learning architectures can reduce computation by 10 times. A 2021 study conducted by Google and the University of California, Berkeley showed the company was able to drastically decrease the carbon emissions of its deep learning Transformer model in just four years. By using more efficient models, optimizing processors, and prioritizing low-carbon resources, data centers reduced the training footprint by more than 700 times, the study showed.

“These drastic overall improvements, as well as their trajectory over time, suggest that extrapolating current parameters to predict future carbon dioxide emissions is fraught with peril,” the researchers concluded. 

Building a backbone for the new AI economy

At the same time, there’s also peril in relying too heavily on tech improvements to solve the AI energy grab problem, said Brian Janous, former vice president of energy at Microsoft.

“I’m not a believer that technology advancements are going to slow this problem down,” he said. 

The AI boom is drawing attention to the ongoing infrastructure bottleneck, said Janous, who is launching a startup aimed at accelerating the buildup of energy infrastructure, and currently sits on the advisory board at LineVision, a transmission monitoring and analytics company.

“This load is coming up faster than utilities can actually build out infrastructure to support it,” Janous said, adding that it’s important to zero in on “the regulatory issues that would prevent them from investing as fast as we need them to invest.”

That likely means policy changes at both a state and federal level, Janous said.

Given the long timelines for both infrastructure and policy progress, he added that short-term solutions are also sorely needed, such as dynamic line ratings and utility-scale storage. 

But these evolving energy demands — and the batchable nature of some of the most compute-intensive applications — may ultimately lead to the rise of a new kind of data center.

New York-based Soluna is building small-footprint data centers optimized to take advantage of excess clean energy; their modular centers are co-located with remote renewables projects, and cater specifically to compute-intensive applications, like training AI models or mining crypto.

Because data centers require 24/7 power and are typically connected directly to the grid, relying solely on renewables is complicated. But that’s much less of an issue for application-specific data centers, said Soluna CEO John Belizaire.

“A lot of these capabilities are inherently batchable,” Belizaire said. “If you’re training an AI model, you can train it for a few days, you can train it for a few months, you can pause that training and pick it back up again.”

This model also has the potential to drive more renewable penetration on the grid, he added, because projects can avoid curtailment and get tax credits for electricity they otherwise wouldn’t have sold.

But a future full of flexible, modular data centers that can expand and contract with energy availability will take time — especially for computing giants that have already invested billions into their own data center infrastructure.

“You won’t see rip-and-replace approaches from the biggest players,” Belizaire said. Instead, we’re more likely to see a slow shift over the next five years as designs like Soluna’s are proven out.

The big question, however, is whether any of these solutions can happen quickly enough to mitigate the looming reality that data center demand is keeping fossil fuels online longer.

“I don’t think there’s a quick fix, per se, but there are things we can do in the interim,” Janous said. “There’s ways we can use the system we have more efficiently.”

Nailing down the data on just how much electricity AI workloads consume is the most immediate step, said Koomey, who is part of a cohort of researchers working to build a report on AI energy use for Congress.

“By the end of the year, we’ll have a reasonable sense of where we are and where we might go,” he said. 

In the meantime, though, “people just need to calm the heck down,” Koomey added. “They’re running around like chickens without heads when, honestly, we do not know what will happen.”

For more on the infrastructure challenges around the coming boom in power demand, including from AI, listen to this Carbon Copy episode: