Prefill and decode do not have to run the same weights. This repo holds prefillers: NVFP4 transformer blocks that are streamed off the SSD one layer at a time, paired to complement low-bit GGUF for decode.

The point is that the two phases want different things. Decode is memory-bound, has to be on device for fast generation and is accelerated by being small. Prefill is compute-bound, so there’s little point in compressing it below NVFP4 (compute-native on Blackwell) as long as it doesn’t occupy much memory, which we achieve via offloading.

Accuracy and time-to-first-token against device weight size

Auxiliary prefillers recover ~50% of the accuracy that aggressive quantization gives up, at close to no extra memory cost. Moreover, past a few thousand tokens, the offloading streaming disappears under compute, so it is not paid for in latency either