AI inference economics is getting even more difficult. Newer models are compute efficient but memory heavy. This is shifting the economics in favor of hyperscalers and away from DIY labs like I have. There is no way you can build a lab in your garage to serve kimi-k3 (after they release their openweights next week). Forget ollama pull kimi-k3...
If you are a financier of hyperscaler (buying their debt), this is good news because DIY and bottom feeders just discoved a BTE (barrier to entry). But if you are a lab want to further research, you are SOL'ed.
Kimi-k3 has 2.8T parameters that all need to be in memory during inferencing. That memory requirement is approximately 1.7TB of DRAM.
It's not just the weights — context eats memory too
That 1.7TB is just the model weights sitting in memory. There is a second memory cost that scales with your traffic: the context window. Every conversation with the model lives in memory as the KV cache — key and value vectors computed for every token, at every layer, that must stay resident in memory for the entire session so the model can attend back to them.
This means your memory bill grows with (a) context length and (b) concurrent users. One long-context session can chew through tens of GB of KV cache. Multiply by thousands of simultaneous users and the KV cache can rival the weights themselves.
So the DIY math is even worse than the table suggests. You don't just need enough memory to hold the model — you need enough to hold every user's conversation at the same time. Hyperscalers spread that cost across a fleet; your garage lab eats it all at once.
| Precision Format | Memory Size | Platform |
|---|---|---|
| MXFP4 (Kimi's own FP4) | ~1.5-2 TB | Multigpu cluster (single rack at least) |
| Q4 Quantization | ~1.0-1.2 TB | Single Enterprise Server with multiple GPUs |
| Aggressive 2-bit | ~1TB | Might fit in a high end laptop, but performance is abysmal and lots of hallucinations |
| BF16/BF32 | ~6-12TB | Multirack solution - Only option is hosted |
- Current consumer and even most enterprise GPUs have limited hardware support for sparse operations.
- Sparse models often require custom kernels and careful integration with inference engines.
- Quality can vary dramatically depending on how aggressively sparsity is applied.
A 8GPU host costs around 700K today, to host the high precision model, we will need 3-5 racks of these hosts (assuming 2 per rack) to get 12-16TB in the cluster. The cost of that is approximately $6M. Add to that cost of power and cooling and we are talking $7-8M. Now imagine you have a great new model optimization, you will need to raise 50M+ just in seed round to get started to support a multi million burn rate.
There is one technical path that could meaningfully change this equation: sparsity.
Sparsity techniques prune weights to zero during or after training, then use specialized kernels that skip calculations and storage for those zeros. In theory, a 2.8T parameter model with 70–90% sparsity could reduce its memory footprint by 3–5× while preserving most of its capability.
If the Kimi-k3 weights are released with high-quality sparsity patterns (or if the community rapidly develops them), the effective memory requirement could drop from 1.7 TB down to the 400–600 GB range. That would bring the model back into reach for well-equipped single-server labs instead of requiring an entire rack.
However, significant challenges remain:
The lab that cracks efficient, high-sparsity inference at this scale will enjoy a massive cost advantage. Until then, the economic headwinds strongly favor those who can amortize millions of dollars of infrastructure across vast workloads — the hyperscalers.
