SvelteBench Visualization

Note: OpenAI thinking models (o3, o4) do not support temperature adjustments. o1-pro models use "medium" reasoning effort setting.

Rank	Model	Score
1	claude-opus-4-20250514 (Anthropic)	88.9%
2	claude-sonnet-4-20250514 (Anthropic)	88.9%
3	x-ai/grok-4 (OpenRouter)	87.8%
4	moonshotai/kimi-k2 (OpenRouter)	84.4%
5	gemini-2.5-pro-preview-03-25 (Google)	83.3%
6	x-ai/grok-3-beta (OpenRouter)	83.3%
7	gemini-2.5-pro-preview-06-05 (Google)	81.1%
8	gemini-2.5-pro-preview-05-06 (Google)	80.0%
9	claude-3-5-haiku-20241022 (Anthropic)	73.3%
10	mistralai/mistral-medium-3 (OpenRouter)	72.2%
11	gemini-2.5-flash-preview-04-17 (Google)	70.0%
12	qwen/qwen3-coder (OpenRouter)	67.8%
13	x-ai/grok-3-mini-beta (OpenRouter)	67.8%
14	meta-llama/llama-4-maverick (OpenRouter)	64.4%
15	qwen/qwen3-235b-a22b-thinking-2507 (OpenRouter)	57.8%
16	claude-3-7-sonnet-20250219 (Anthropic)	56.7%
17	qwen/qwen3-235b-a22b-07-25 (OpenRouter)	55.6%
18	mistralai/devstral-medium (OpenRouter)	52.2%
19	deepseek/deepseek-r1-0528 (OpenRouter)	48.9%
20	z-ai/glm-4-32b (OpenRouter)	46.7%
21	qwen/qwen3-30b-a3b (OpenRouter)	34.4%
22	o3-2025-04-16 (OpenAI)	30.0%
23	chatgpt-4o-latest (OpenAI)	25.6%
24	gpt-4.1-2025-04-14 (OpenAI)	22.2%
25	gpt-4o-2024-08-06 (OpenAI)	17.8%
26	o3-mini-2025-01-31 (OpenAI)	15.6%
27	meta-llama/llama-4-scout (OpenRouter)	15.6%
28	o4-mini-2025-04-16 (OpenAI)	13.3%
29	mistralai/devstral-small (OpenRouter)	13.3%
30	gemma-3-27b-it (Google)	11.1%
31	gpt-4.1-nano-2025-04-14 (OpenAI)	11.1%
32	o1-pro-2025-03-19 (OpenAI)	11.1%
33	google/gemma-3n-e4b-it (OpenRouter)	8.9%
34	moonshotai/kimi-dev-72b:free (OpenRouter)	3.3%
35	gpt-4.1-mini-2025-04-14 (OpenAI)	1.1%

Anthropic

▼

claude-3-5-haiku-20241022

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	⚠️ PARTIAL	0.6000	1.0000	6/10	4	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

claude-3-7-sonnet-20250219

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	⚠️ PARTIAL	0.1000	1.0000	1/10	9	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

claude-opus-4-20250514

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
snippets	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests

claude-sonnet-4-20250514

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
snippets	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests

Google

▼

gemini-2.5-flash-preview-04-17

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.6000	1.0000	6/10	4	View Prompt View Tests
each	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
effect	⚠️ PARTIAL	0.7000	1.0000	7/10	6	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
snippets	⚠️ PARTIAL	0.1000	1.0000	1/10	9	View Prompt View Tests

gemini-2.5-pro-preview-03-25

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
snippets	⚠️ PARTIAL	0.5000	1.0000	5/10	5	View Prompt View Tests

gemini-2.5-pro-preview-05-06

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	⚠️ PARTIAL	0.8000	1.0000	8/10	4	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
snippets	⚠️ PARTIAL	0.6000	1.0000	6/10	4	View Prompt View Tests

gemini-2.5-pro-preview-06-05

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	⚠️ PARTIAL	0.9000	1.0000	9/10	2	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
snippets	⚠️ PARTIAL	0.4000	1.0000	4/10	6	View Prompt View Tests

gemma-3-27b-it

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
derived-by	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

OpenAI

▼

chatgpt-4o-latest

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.7000	1.0000	7/10	3	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.1000	1.0000	1/10	9	View Prompt View Tests
each	⚠️ PARTIAL	0.4000	1.0000	4/10	6	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	⚠️ PARTIAL	0.1000	1.0000	1/10	9	View Prompt View Tests

gpt-4.1-2025-04-14

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.7000	1.0000	7/10	3	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
derived-by	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
each	⚠️ PARTIAL	0.2000	1.0000	2/10	9	View Prompt View Tests
effect	⚠️ PARTIAL	0.1000	1.0000	1/10	10	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	19	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

gpt-4.1-mini-2025-04-14

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
derived-by	❌ FAIL	0.0000	0.0000	0/10	12	View Prompt View Tests
each	⚠️ PARTIAL	0.1000	1.0000	1/10	10	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	16	View Prompt View Tests
hello-world	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	28	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

gpt-4.1-nano-2025-04-14

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	❌ FAIL	0.0000	0.0000	0/10	14	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
derived-by	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

gpt-4o-2024-08-06

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	14	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.4000	1.0000	4/10	14	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/10	11	View Prompt View Tests
effect	⚠️ PARTIAL	0.2000	1.0000	2/10	14	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	14	View Prompt View Tests

o1-pro-2025-03-19

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	❌ FAIL	0.0000	0.0000	0/1	1	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/1	1	View Prompt View Tests
derived-by	❌ FAIL	0.0000	0.0000	0/1	3	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/1	1	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/1	1	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	1/1	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/1	1	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/1	1	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/1	1	View Prompt View Tests

o3-2025-04-16

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.6000	1.0000	6/10	4	View Prompt View Tests
derived	⚠️ PARTIAL	0.1000	1.0000	1/10	11	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.2000	1.0000	2/10	8	View Prompt View Tests
each	⚠️ PARTIAL	0.6000	1.0000	6/10	4	View Prompt View Tests
effect	⚠️ PARTIAL	0.2000	1.0000	2/10	11	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

o3-mini-2025-01-31

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.3000	1.0000	3/10	13	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	11	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.1000	1.0000	1/10	9	View Prompt View Tests
each	⚠️ PARTIAL	0.1000	1.0000	1/10	9	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	17	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

o4-mini-2025-04-16

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.1000	1.0000	1/10	12	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	12	View Prompt View Tests
derived-by	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
each	⚠️ PARTIAL	0.2000	1.0000	2/10	8	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	11	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

OpenRouter

▼

deepseek/deepseek-r1-0528

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.3000	1.0000	3/10	22	View Prompt View Tests
derived	⚠️ PARTIAL	0.6000	1.0000	6/10	6	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
each	⚠️ PARTIAL	0.2000	1.0000	2/10	10	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	⚠️ PARTIAL	0.4000	1.0000	4/10	6	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

google/gemma-3n-e4b-it

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	❌ FAIL	0.0000	0.0000	0/10	19	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
derived-by	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.8000	1.0000	8/10	2	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

meta-llama/llama-4-maverick

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.8000	1.0000	8/10	2	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

meta-llama/llama-4-scout

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
derived	⚠️ PARTIAL	0.2000	1.0000	2/10	8	View Prompt View Tests
derived-by	❌ FAIL	0.0000	0.0000	0/10	11	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/10	11	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.3000	1.0000	3/10	11	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

mistralai/devstral-medium

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	⚠️ PARTIAL	0.8000	1.0000	8/10	4	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
each	⚠️ PARTIAL	0.1000	1.0000	1/10	9	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	19	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

mistralai/devstral-small

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.7000	1.0000	7/10	6	View Prompt View Tests
derived	⚠️ PARTIAL	0.1000	1.0000	1/10	13	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.1000	1.0000	1/10	13	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.3000	1.0000	3/10	7	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

mistralai/mistral-medium-3

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	⚠️ PARTIAL	0.8000	1.0000	8/10	2	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	16	View Prompt View Tests
props	⚠️ PARTIAL	0.8000	1.0000	8/10	2	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

moonshotai/kimi-dev-72b:free

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
derived-by	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.3000	1.0000	3/10	8	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

moonshotai/kimi-k2

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	22	View Prompt View Tests
props	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
snippets	⚠️ PARTIAL	0.7000	1.0000	7/10	4	View Prompt View Tests

qwen/qwen3-235b-a22b-07-25

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	⚠️ PARTIAL	0.6000	1.0000	6/10	4	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.8000	1.0000	8/10	2	View Prompt View Tests
each	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
effect	⚠️ PARTIAL	0.4000	1.0000	4/10	7	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	22	View Prompt View Tests
props	⚠️ PARTIAL	0.4000	1.0000	4/10	9	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

qwen/qwen3-235b-a22b-thinking-2507

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.8000	1.0000	8/10	2	View Prompt View Tests
derived	⚠️ PARTIAL	0.8000	1.0000	8/10	3	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.6000	1.0000	6/10	4	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	⚠️ PARTIAL	0.8000	1.0000	8/10	3	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	16	View Prompt View Tests
props	⚠️ PARTIAL	0.2000	1.0000	2/10	11	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	14	View Prompt View Tests

qwen/qwen3-30b-a3b

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.6000	1.0000	6/10	7	View Prompt View Tests
derived	⚠️ PARTIAL	0.3000	1.0000	3/10	8	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.3000	1.0000	3/10	9	View Prompt View Tests
each	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
effect	⚠️ PARTIAL	0.3000	1.0000	3/10	11	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.7000	1.0000	7/10	5	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	22	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

qwen/qwen3-coder

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.3000	1.0000	3/10	21	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	⚠️ PARTIAL	0.9000	1.0000	9/10	2	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	37	View Prompt View Tests
props	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

x-ai/grok-3-beta

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
snippets	⚠️ PARTIAL	0.5000	1.0000	5/10	5	View Prompt View Tests

x-ai/grok-3-mini-beta

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.6000	1.0000	6/10	7	View Prompt View Tests
each	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
effect	⚠️ PARTIAL	0.7000	1.0000	7/10	6	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	40	View Prompt View Tests
props	⚠️ PARTIAL	0.8000	1.0000	8/10	2	View Prompt View Tests
snippets	⚠️ PARTIAL	0.1000	1.0000	1/10	11	View Prompt View Tests

x-ai/grok-4

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.9000	1.0000	9/10	3	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
snippets	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests

z-ai/glm-4-32b

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.8000	1.0000	8/10	4	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
effect	⚠️ PARTIAL	0.2000	1.0000	2/10	12	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	25	View Prompt View Tests
props	⚠️ PARTIAL	0.3000	1.0000	3/10	7	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

SvelteBench Visualization

🏆 Top Models Leaderboard

Anthropic

claude-3-5-haiku-20241022

claude-3-7-sonnet-20250219

claude-opus-4-20250514

claude-sonnet-4-20250514

Google

gemini-2.5-flash-preview-04-17

gemini-2.5-pro-preview-03-25

gemini-2.5-pro-preview-05-06

gemini-2.5-pro-preview-06-05

gemma-3-27b-it

OpenAI

chatgpt-4o-latest

gpt-4.1-2025-04-14

gpt-4.1-mini-2025-04-14

gpt-4.1-nano-2025-04-14

gpt-4o-2024-08-06

o1-pro-2025-03-19

o3-2025-04-16

o3-mini-2025-01-31

o4-mini-2025-04-16

OpenRouter

deepseek/deepseek-r1-0528

google/gemma-3n-e4b-it

meta-llama/llama-4-maverick

meta-llama/llama-4-scout

mistralai/devstral-medium

mistralai/devstral-small

mistralai/mistral-medium-3

moonshotai/kimi-dev-72b:free

moonshotai/kimi-k2

qwen/qwen3-235b-a22b-07-25

qwen/qwen3-235b-a22b-thinking-2507

qwen/qwen3-30b-a3b

qwen/qwen3-coder

x-ai/grok-3-beta

x-ai/grok-3-mini-beta

x-ai/grok-4

z-ai/glm-4-32b