The quoted numbers are the interesting part. 380 GB just for weights plus several GB of context per user means this only pays off at Revolut’s scale. What stood out to me is that the article credits the biggest gains to NVFP4 quantization, splitting prefill and decode, and context tuning rather than adding GPUs. Those tricks carry over to much smaller setups too.

Rule of thumb that holds up well: model file size + room for context + a bit for the process.

• 7-8B at 4-bit: ~4-5 GB file, so 8 GB total is comfortable
• 13B at 4-bit: plan ~12 GB
• 70B: not a CPU thing, don’t bother

The one people forget is context. Cranking num_ctx (or OLLAMA_CONTEXT_LENGTH) up because the model “supports 128k” eats memory fast, and the usual symptom is a model that loads fine and then dies halfway through a long answer.

On CPU, expect roughly 5-15 tok/s for a 4-bit 7-8B on decent cores. Memory bandwidth matters more than core count after a point.