Leaderboard
Generation speed ranked across accelerators — for a given model. Beautiful, not a spreadsheet.
Llama 3.2 1B Instruct 32Meta Llama 3.1 8B Instruct 14Qwen2.5 14B Instruct 12DeepSeek R1 671B 2Goliath 120B 2Qwen3 235B 2qwen3:14b 2
1
NVIDIA H100
80 GB ·LocalScore
120tok/s
—
2
NVIDIA A100 80GB
80 GB ·LocalScore
110tok/s
—
3
NVIDIA RTX 3090 Ti
24 GB ·LocalScore
110tok/s
—
4
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
95 GB ·LocalScore
103tok/s
—
5
NVIDIA RTX 3090
24 GB ·LocalScore
103tok/s
0.31 tok/s/W
6
NVIDIA RTX A6000
48 GB ·LocalScore
90.5tok/s
—
7
NVIDIA RTX 4090
24 GB ·LocalScore
90.3tok/s
—
8
NVIDIA GeForce RTX 3070 Ti
8 GB ·LocalScore
83.2tok/s
—
9
NVIDIA RTX 4080
16 GB ·LocalScore
78.7tok/s
—
10
NVIDIA RTX 5090
32 GB ·LocalScore
66.3tok/s
—
11
Apple M4 Max
128 GB ·LocalScore
51.6tok/s
1.78 tok/s/W
12
NVIDIA RTX 6000 Ada Generation
48 GB ·LocalScore
51.3tok/s
—
13
NVIDIA GeForce RTX 4060 Ti
16 GB ·LocalScore
49.3tok/s
0.37 tok/s/W
14
AMD Radeon RX 6650 XT
8 GB ·LocalScore
25.9tok/s
—