Leaderboard
Generation speed ranked across accelerators — for a given model. Beautiful, not a spreadsheet.
Llama 3.2 1B Instruct 32Meta Llama 3.1 8B Instruct 14Qwen2.5 14B Instruct 12DeepSeek R1 671B 2Goliath 120B 2Qwen3 235B 2qwen3:14b 2
1
NVIDIA L40S
44 GB ·LocalScore
355tok/s
—
2
NVIDIA RTX 3090 Ti
24 GB ·LocalScore
354tok/s
—
3
NVIDIA H100
80 GB ·LocalScore
335tok/s
—
4
NVIDIA RTX 3090
24 GB ·LocalScore
330tok/s
1.14 tok/s/W
5
NVIDIA RTX A6000
48 GB ·LocalScore
315tok/s
—
6
NVIDIA A100 80GB
80 GB ·LocalScore
308tok/s
—
7
NVIDIA GeForce RTX 3070 Ti
8 GB ·LocalScore
297tok/s
—
8
Quadro RTX 6000
23 GB ·LocalScore
253tok/s
—
9
NVIDIA Tesla V100
32 GB ·LocalScore
248tok/s
—
10
NVIDIA RTX 4080
16 GB ·LocalScore
247tok/s
—
11
NVIDIA GeForce RTX 4060 Ti
16 GB ·LocalScore
218tok/s
1.95 tok/s/W
12
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
95 GB ·LocalScore
215tok/s
—
13
NVIDIA RTX 4090
24 GB ·LocalScore
207tok/s
—
14
NVIDIA GeForce RTX 2070 Super
8 GB ·LocalScore
189tok/s
—
15
Apple M4 Max
128 GB ·LocalScore
184tok/s
5.2 tok/s/W
16
NVIDIA RTX 5090
32 GB ·LocalScore
170tok/s
—
17
NVIDIA RTX 6000 Ada Generation
48 GB ·LocalScore
131tok/s
—
18
AMD Radeon RX 6650 XT
8 GB ·LocalScore
109tok/s
—
19
NVIDIA RTX PRO 6000 Blackwell Server Edition
95 GB ·LocalScore
106tok/s
—
20
NVIDIA H200
141 GB ·LocalScore
98.5tok/s
—
21
Apple M1
8 GB ·LocalScore
80.4tok/s
3.56 tok/s/W
22
AMD EPYC 7352 24-Core Processor (znver2)
126 GB ·LocalScore
14.3tok/s
—