Latency is one of the easiest cloud gaming problems to understand. Press a button, and that command has to reach a remote server. The game reacts, renders the next frame, and sends it back to your screen. The longer that process takes, the more noticeable the delay can become.
For companies operating GPU infrastructure, latency isn’t only about how an experience responds. A GPU can be installed, powered, and available, yet still wait for data, a network transfer, or synchronization with other accelerators. Those delays can reduce how much useful work the same infrastructure produces.
Boosteroid recently argued that latency is also an infrastructure economics problem because delays can reduce GPU utilization. The company connects that argument to delays involving networking, storage, synchronization, and other dependencies in distributed workloads. That raises a broader question that reaches beyond cloud gaming. How much useful compute can operators actually get from the GPUs they’ve already installed?
Installed GPU Count Doesn’t Equal Useful Compute
A GPU count tells you how many accelerators are available, not how productively they’re being used. Cloud gaming provides an easy example because every interaction depends on remote compute responding quickly enough to preserve the connection between an input and what appears on screen.
When NVIDIA detailed its Blackwell GeForce NOW upgrade in 2025, it said the service would support click-to-pixel response times as low as 30 milliseconds. NVIDIA also said most supported regions would see sub-30 millisecond network latency. Those figures are specific to GeForce NOW, but they illustrate how much engineering attention a commercial cloud gaming service gives to response time.
Boosteroid’s argument looks at the same issue from the operator side. If an accelerator has to wait for data or another dependency, the deployed infrastructure produces less useful work during that delay. The economic question isn’t simply how many GPUs are installed. It is how effectively those accelerators can be kept productive.
Cloud gaming and AI don’t have the same latency requirements. Cloud gaming needs the remote response to arrive quickly enough that controls remain responsive. Large AI workloads have different timing targets and dependencies, but accelerator count alone still doesn’t describe their productive output.
AI Training Can Leave GPUs Waiting
Large AI training jobs can spread work across many GPUs that need to exchange data and coordinate before the next step can continue. Meta describes the network as being in the critical path for both AI training and inference. In synchronized training, the slowest transfer can set the pace for the wider job.
That means a network delay can affect more than the accelerator directly involved in the transfer. Other GPUs may be ready to continue, yet synchronization can force them to wait. Meta says even modest network delays can strand meaningful compute capacity in large distributed workloads.
Storage can create another bottleneck. Meta’s AI storage work identifies storage delays as a major cause of GPU stalls. If one accelerator is waiting for data and the workload depends on synchronized progress, the slowdown can affect other GPUs tied to the same training step.
Production research presented at USENIX NSDI 2026 gives the problem a useful reality check. EROICA operated across production clusters totalling roughly 100,000 GPUs. The researchers reported throughput improvements of about 20% to 100% after diagnosed performance problems were corrected in the affected training workloads.
Those results apply to workloads that were already experiencing serious performance problems. They aren’t evidence that every GPU cluster has a similar amount of unused capacity. One 3,400-GPU video-model training case improved from 7,173 to 9,644 samples per day, a 34% end-to-end increase, after several issues were corrected.
That case also shows why latency can’t be treated as the only problem. The fixes included network throughput, a network interface issue, memory and data-loading problems, and workload imbalance. Better output came from addressing several constraints around the accelerators rather than simply adding more GPUs.
AI Inference Has a Latency and Throughput Tradeoff
Inference creates a different economic problem. Instead of coordinating a training job across a large group of accelerators, an inference service has to answer requests quickly enough for its intended use and process enough requests to use its GPU capacity efficiently.
Google Cloud describes this through a latency and throughput tradeoff. With a fixed budget for accelerator capacity, an operator can trade lower latency against higher throughput. Software and infrastructure improvements can move that performance frontier, allowing the same budget to produce better results.
That puts efficiency directly into the economics of serving AI models. Two deployments with similar accelerator capacity can produce different latency and throughput results depending on how effectively the inference stack uses that capacity. Poor utilization therefore reduces the useful output available from infrastructure that has already been paid for and deployed.
A game stream prioritizes interactive response. An AI service may balance response time against request volume, model size, and GPU efficiency. In either case, accelerator count alone doesn’t determine useful output.
Useful Compute Matters More Than GPU Count
This is the broader value in Boosteroid’s argument. Buying more GPUs can increase theoretical capacity, yet accelerator count says little about the delays surrounding those GPUs. Networking, storage, synchronization, software, and workload behaviour all affect how much productive work comes from installed capacity.
For cloud gaming, latency appears immediately as a response on screen. In AI infrastructure, delays can reduce throughput or utilization from capacity that’s already installed. In both cases, the more useful question isn’t only how much GPU capacity exists. It’s how much useful compute that capacity can actually produce.
As always, remember to follow us on our social media platforms (e.g., Threads, X (Twitter), Bluesky, YouTube, and Facebook) to stay up-to-date with the latest news. This website contains affiliate links. We may receive a commission when you click on these links and make a purchase, at no extra cost to you. We are an independent site, and the opinions expressed here are our own.
