Roughly 30,000 data centers dot the landscape in the United States and Europe.
These computer facilities serve functions as varied as modeling and simulation, cloud-based applications, artificial intelligence and weather forecasting. They are the brains of search engines and social media platforms, and they help engineers design the automobiles and airplanes of the future. In short, data centers form the basis for most modern computing.
They also consume a lot of power. Most of these data centers use between 10 megawatts (MW) to 200 MW of power, running all day and all night. For context, 1 MW, or 1,000 kilowatts (kW), can power about 400 to 900 homes. Supercomputers require giant backup generators and expensive battery systems in case of a power outage. In energy-starved locations, competition for scarce electrons is driving up the cost of electricity for data centers and households alike.
Now, INL researchers are exploring a novel approach to building data centers. By connecting supercomputers in fully equipped shipping containers to microgrids powered by microreactors and renewables, data centers could become carbon-free, more flexible, mobile and less expensive.
These supercomputers-in-a-box could someday serve as the basis for similar systems on the moon and Mars. For now, researchers must answer some tough questions, especially how to run these data centers when power is limited.
The Collaborative Computing Center
Idaho National Laboratory’s Collaborative Computing Center has a lot in common with today’s data centers. Its Sawtooth supercomputer uses 100,000 processors to churn through some of the world’s most complicated modeling and simulation jobs. This machine, along with INL’s other supercomputers, are funded by the Nuclear Science User Facilities.
Sawtooth and its infrastructure — including 2.4 MW of backup batteries and a massive cooling system that pumps 1,200 gallons of water per minute — takes up an entire wing of the building. The computer alone consumes an average of 1 MW of power.
Behind the center, two house-sized buildings hold more of the center’s infrastructure: a 3 MW backup generator and giant cooling system over a large reservoir of water.
Tucked between these buildings sits an innocuous-looking green shipping container.
Mobile supercomputer
This shipping container holds almost everything a data center might need: its own battery backup system, power conditioning, HVAC, lighting, fire suppression system, a single-source main panel feed, four racks for processors, eight in-rack power distribution units and vibration isolation. It’s even weather-proof as long as the doors are kept shut. The container was loaned to INL by World Wide Technology Inc., a technology company based in St. Louis, Missouri.
Power can come from a variety of sources, including a normal grid or a microgrid connected to various renewable energy sources or a microreactor. The platform will host a supercomputer roughly 1/100th the size of Sawtooth and will draw up to 50 kW in power (including cooling).
Figuring out how to provide that power presents an exciting opportunity, says INL researcher and High Performance Computing department manager Matt Anderson. “Since the supercomputer is powered through one panel, the electricity use can be monitored very carefully to determine how much power it takes to run each job,” he said. “That’s a luxury we don’t have with Sawtooth.”
Knowing each job’s power requirements is important. Eventually, these data centers might run off microgrids powered by a combination of carbon-free renewables and microreactors. Microgrids are small, smart electrical grids with local power sources that are either isolated or connected to the larger grid. Microreactors are nuclear reactors that would be built in factories that are small enough to be shipped around the world.
With microgrids and microreactors, multiple supercomputers-in-a-box could be dropped in the middle of a desert or stacked on top of one another on a cargo ship to create a carbon-free data center almost anywhere.
Managing power consumption by managing jobs
But to realize that goal, managing power consumption is critical.
“The software to run most data centers was designed in the ‘90s with the assumption that there’s plenty of power,” Anderson said. “Here, researchers will test what happens when power for an isolated data center is in short supply and must be carefully managed.”
“We want to know if you can eliminate most of the need for batteries and backup generators and run an isolated data center off a microgrid,” Anderson continued.
This is a complex engineering challenge — the power quality from a microgrid isn’t as good because it is hard to control frequency and voltage — but it’s also a software challenge.
Because the waiting line for most jobs on Sawtooth is on a first-come, first-served basis, Anderson and his colleagues are trying to find a way to manage jobs to match the available power supply.
A job management system that relies on machine learning and artificial intelligence will consider how weather and daylight will affect wind and solar generation and queue the jobs accordingly. That’s why it’s important to know exactly how much power each job takes.
Testing the system
Over the coming months, Anderson and his colleagues will build the computer and test the system using several different power sources. The first tests will occur at the Collaborative Computing Center using power from the regular electric grid.
There, researchers will test the scheduling software, the weather prediction component and the supercomputer’s ability to switch into a “suspend state,” where the jobs underway are essentially frozen in case of an outage.
The suspend state capability is a big deal for supercomputers. Often, even with backup batteries and generators, the power isn’t restored fast enough to save a given job’s progress. That loss of data can cost hundreds of thousands of dollars.
Researchers will then relocate the container behind INL’s Energy Systems Laboratory where the mobile supercomputer will be connected to INL’s microgrid-in-a-box, a portable microgrid with control features and specifications that can manage about 250 kW of electricity from batteries, solar and wind. “We will be repeating the exact same experiments, but this time with the real microgrid either tied to the regular grid or islanded,” Anderson said.
Eventually, researchers plan to move the container to INL’s Transient Reactor Test Facility. There, it may connect to the Microreactor Applications Research Validation and Evaluation Project (MARVEL), a 100 kW thermal microreactor currently under construction. With MARVEL, researchers will evaluate the potential of the supercomputer as a carbon-free resource under real-world conditions with a full-scale microgrid and wind, solar and nuclear generation.
Real-world applications
While the main goal of the project is science — gathering data and developing software to optimize carbon-free data centers — there are lots of real-world uses for a mobile supercomputer. These include military bases and remote industrial operations such as mines. Since the supercomputer is mobile, it could be moved to the site of a large event, such as a sporting event, to help provide 5G support.
At these types of locations, it may not be possible to build a conventional data center and there may not even be a standard power grid available for connection. The experiments at INL could allow much greater computing capacity at remote and extreme locations than before.
For Anderson and his colleagues, the mobile supercomputer has another exciting application: space. NASA has already sent two versions of a supercomputer, the Spaceborne Computer, to the International Space Station.
Supercomputers for proposed settlements on the moon or Mars could be powered by a similar combination of technologies as the mobile supercomputer: a microreactor, microgrid and possibly solar, under tightly controlled power allocation conditions.
“For the data center of the future, we want to know if we can get to net-zero carbon emissions,” Anderson said. “We want high reliability and high utilization. We want to eliminate risk and provide a pathway forward for people to follow.”
