If you've shopped GPU inference recently, you've seen the sticker shock. An H100 reserved on a major cloud is north of $4/hr. Spot is cheaper but unpredictable. A dedicated 8-GPU box runs into the tens of thousands per month. And that's before egress, support, or any of the value-added tooling.
That number doesn't come from nowhere. It reflects three real costs the cloud has to recover. Each one creates an opening for a distributed model to do better.
Cost driver one: utilization.
A reserved GPU is billed whether you use it or not. The cloud is selling you the right to a piece of silicon for a window of time, and they need the average customer to use it enough that the rate covers their costs. Most customers don't. Internal data from operators we've talked to puts average enterprise GPU utilization at 15–40% across the day, with single-digit utilization overnight and on weekends.
You're paying the per-hour rate for 100% of the hours, but only getting useful work out of a fraction. The rest is amortized into the price you see on the page.
Cost driver two: CapEx.
An H100 retails around $30,000 today. An 8-GPU node with networking and host hardware is comfortably $300,000 plus install. The cloud provider pays that up front and recovers it over the GPU's economic life — usually 3–5 years. They also have to fund the depreciation of the previous generation that's now cheap, and the buffer for the next generation that isn't shipping yet.
Add power, cooling, real estate, and operations staff, and you're looking at a fully-loaded TCO of $5–7/hr per H100 just to break even at full utilization. The published rate has to clear that plus the utilization gap from above.
Cost driver three: the real-time premium.
This is the one most people miss. When you provision GPU capacity, you're paying for it to be warm and waiting. The cloud has to keep the hardware online, the model loaded, and the inference pipeline ready to serve a request in milliseconds — even if no request arrives.
That standby state has a real cost. It's the same reason on-demand cloud pricing is 4–6x spot pricing: you're paying for guaranteed availability, not for compute that ran. And for many workloads, you don't need that guarantee. A document-processing pipeline that runs nightly does not care whether each job starts in 50ms or 5 seconds. A dataset enrichment job that takes four hours doesn't care if it takes four hours and three minutes.
If your workload tolerates a few seconds of queue time, you are massively overpaying for it on a real-time-priced GPU.
What changes with a distributed marketplace.
A distributed inference network rewires all three of those drivers.
Utilization. Idle GPUs are everywhere — in research labs, gaming PCs, small colos, and over-provisioned enterprise clusters. Their owners already paid the CapEx; the marginal cost of running a job on them is power, network, and a small operator margin. A marketplace lets those owners sell that idle capacity at a price that reflects marginal cost, not amortized CapEx. The aggregate effect is to find the GPUs that would have been at 5% utilization and lift them to 60%, with the price savings flowing to the customer.
CapEx. The marketplace operator doesn't own the GPUs. We don't carry the depreciation, we don't fund the next-gen buffer, and we don't have to amortize a fleet. We charge a small platform fee on each job — transaction infrastructure, scheduling, billing, support — and the rest goes to the operator who owns the silicon.
Real-time premium. An asynchronous job queue is a different shape of product. Submit a job, get a job ID back, and pick up the result when it's done. No warm capacity reserved for you. The compute spins up when the scheduler matches your job to an available worker, runs, and releases. You pay only for the work that ran, at the rate of compute that wasn't going to be used otherwise.
The numbers, roughly.
Take a Llama 3.1 8B job that produces 1,000 output tokens. On a major cloud's serverless inference product, that runs roughly $0.10–$0.20 today, including the real-time premium. The same job on MicroDC.ai bills around $0.011 — a base fee plus per-token rate — about 10x cheaper.
The gap widens for larger models. A Llama 3.3 70B job at the same length is north of $1 on real-time serverless and well under $0.10 on a distributed network. For container jobs — arbitrary Docker images on GPU — the gap is even larger because the cloud's container-on-GPU products carry an additional reservation premium.
Where this doesn't work.
Worth being honest: a distributed marketplace is the wrong shape for some workloads.
- Sub-second user-facing latency. If your product needs to start streaming tokens to a user in 200ms, a queued asynchronous job is not your tool. Real-time inference is what you want.
- Mission-critical real-time systems. If a dropped job means a real-world failure, you want the cloud's SLA, not a distributed pool.
- Strict locality requirements. If data sovereignty rules say compute must run in a specific jurisdiction, a marketplace with operators in many locations needs careful filtering — possible, but more work.
For everything else — document processing, dataset enrichment, content generation, research workloads, automation pipelines, overnight batches — the math heavily favors the distributed model.
The takeaway.
"GPU inference is expensive" is not a fact about GPUs. It's a consequence of three pricing decisions made by hyperscalers: bundling unused capacity into the rate, recovering CapEx through that rate, and including a real-time SLA you may not need. Take any one of those out and the price falls by half. Take all three out and you get the order-of-magnitude reduction we're seeing in production.
If your workload tolerates queue time — and most batch workloads do — you should be running it somewhere that doesn't make you pay for what you're not using.