Decoding one token means reading every active weight from memory once. That work is
memory-bandwidth-bound, not compute-bound, for the batch-size-one,
one-token-at-a-time pattern local inference almost always runs: the GPU spends most of
its time waiting on memory, not doing math. So the number that predicts decode speed is
roughly:
tokens/sec ≈ effective memory bandwidth ÷ bytes read per token
"Bytes read per token" is the model's active parameter count at its
quantization — not its total parameter count. This is the detail that trips up
mixture-of-experts (MoE) models specifically: a MoE model with 30B total parameters but
only 3B active per token reads roughly a tenth as much memory per token as a 30B
dense model would, and decodes proportionally faster. Two models can be labeled
with the same total size and behave completely differently at inference time depending on
this one distinction.
This also works as a sanity check in the other direction: given a card's real bandwidth
and a model's claimed active-parameter count and quantization, you can predict roughly
what tok/s it should decode at. If a model implies a bandwidth above what the
hardware is physically capable of, the metadata is wrong — not the hardware.
One row in the table below is not a vendor spec at all: the DGX Spark's bandwidth figure
is this project's own measured effective decode bandwidth, taken on the exact machine
this benchmark runs on — not the vendor's theoretical-peak number every other row
carries. It is marked measured in the table so it
is never read as the same kind of figure as its neighbors.
