Leaderboard
Generation speed ranked across accelerators — for a given model. Beautiful, not a spreadsheet.
Llama 3.2 1B Instruct 32Meta Llama 3.1 8B Instruct 14Qwen2.5 14B Instruct 12DeepSeek R1 671B 2Goliath 120B 2Qwen3 235B 2qwen3:14b 2
1
NVIDIA H100
80 GB ·LocalScore
72.2tok/s
—
2
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
95 GB ·LocalScore
70.8tok/s
—
3
NVIDIA A100 80GB
80 GB ·LocalScore
66.3tok/s
—
4
NVIDIA RTX 3090 Ti
24 GB ·LocalScore
64.2tok/s
—
5
NVIDIA RTX 3090
24 GB ·LocalScore
59.9tok/s
0.18 tok/s/W
6
NVIDIA RTX A6000
48 GB ·LocalScore
52tok/s
—
7
NVIDIA RTX 4090
24 GB ·LocalScore
49.1tok/s
—
8
NVIDIA RTX 4080
16 GB ·LocalScore
46.2tok/s
—
9
NVIDIA RTX 5090
32 GB ·LocalScore
45.5tok/s
—
10
NVIDIA RTX 6000 Ada Generation
48 GB ·LocalScore
28.1tok/s
—
11
NVIDIA GeForce RTX 4060 Ti
16 GB ·LocalScore
26.9tok/s
0.2 tok/s/W
12
NVIDIA RTX PRO 6000 Blackwell Server Edition
95 GB ·LocalScore
17.9tok/s
—