Data Centers:  What will they look like in 20 years?  Apple now has the MLX Distributed, which allows multiple Apple Silicon computers to work together on AI workloads: a one-trillion-parameter AI model running across four Mac Studios.  That works because the model is stored in compressed form, and only a small part of it is actually used to answer any one question.

Four relatively small computers.
Roughly 2 TB of unified memory.
A trillion-parameter model.
Sitting under a desk.  Wow!

That doesn’t mean four Macs replace an AI data center, but: 

Two concepts:  Training and Inference

Training — building the model. This requires enormous computing resources and is likely to remain concentrated in large data centers.

Inference — using the finished model. Every time you ask an AI a question and it generates an answer, that’s inference.

Medical school happens once. Office visits happen millions of times. Once the model is trained, it can answer billions of questions. Over time, inference becomes most of the computing moved to local hardware.

Much of that inference could eventually migrate to local servers. A business, university, hospital, engineering firm, government agency or school system could potentially operate substantial AI capability locally — using its own information, behind its own firewall, without sending every request to a hyperscale data center and paying for every token.

A foundation model may be trained periodically. It can subsequently answer billions of requests. And it is that second workload — inference — where increasingly capable local machines could begin changing the equation.

There is a second limit, and it is not storage;  it is speed.  A local machine can hold a very large model affordably.  What it cannot do is answer as fast, or for as many people at once, as a data center can.  That is what sorts the work.  Jobs that serve a few people at a time, where privacy matters more than a few extra seconds, run well locally. 

We’ve seen this movie before.  Computing began largely as Mainframe to terminal.  The PC didn’t eliminate the data center. It moved an enormous amount of computing out of it.

So when we forecast AI data-center electricity demand 10 or 20 years into the future, I think we should be very careful about simply extrapolating today’s architecture.

Local AI processing will not eliminate data centers. But it raises a very important question: What percentage of tomorrow’s AI workload actually needs to be in one?

John May is a principal of GreenPlex (greenplex.ai) and a founder of Greenswell Growers, a controlled-environment agriculture (CEA) farm near Richmond, Virginia.

 

#AI #Inference #DataCenters #EdgeAI #AppleSilicon #AIInfrastructure #EnergyDemand