SvelteBench Visualization

Note: OpenAI thinking models (o3, o4) do not support temperature adjustments. o1-pro models use "medium" reasoning effort setting.

← Back to All Results

Anthropic

▼

claude-3-5-haiku-20241022

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	⚠️ PARTIAL	0.6000	1.0000	6/10	4	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

claude-3-7-sonnet-20250219

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	⚠️ PARTIAL	0.1000	1.0000	1/10	9	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

Google

▼

gemini-2.5-flash-preview-04-17

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.6000	1.0000	6/10	4	View Prompt View Tests
each	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
effect	⚠️ PARTIAL	0.7000	1.0000	7/10	6	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
snippets	⚠️ PARTIAL	0.1000	1.0000	1/10	9	View Prompt View Tests

gemini-2.5-pro-preview-03-25

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
derived-by	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
each	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
effect	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
snippets	⚠️ PARTIAL	0.5000	1.0000	5/10	5	View Prompt View Tests

OpenAI

▼

gpt-4o-2024-08-06

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	14	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.4000	1.0000	4/10	14	View Prompt View Tests
each	❌ FAIL	0.0000	0.0000	0/10	11	View Prompt View Tests
effect	⚠️ PARTIAL	0.2000	1.0000	2/10	14	View Prompt View Tests
hello-world	✅ PASS	1.0000	1.0000	10/10	0	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	14	View Prompt View Tests

o3-mini-2025-01-31

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.3000	1.0000	3/10	13	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	11	View Prompt View Tests
derived-by	⚠️ PARTIAL	0.1000	1.0000	1/10	9	View Prompt View Tests
each	⚠️ PARTIAL	0.1000	1.0000	1/10	9	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	17	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests

o4-mini-2025-04-16

▼

Test	Status	pass@1	pass@10	Passing Samples	Errors	Actions
counter	⚠️ PARTIAL	0.1000	1.0000	1/10	12	View Prompt View Tests
derived	❌ FAIL	0.0000	0.0000	0/10	12	View Prompt View Tests
derived-by	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
each	⚠️ PARTIAL	0.2000	1.0000	2/10	8	View Prompt View Tests
effect	❌ FAIL	0.0000	0.0000	0/10	11	View Prompt View Tests
hello-world	⚠️ PARTIAL	0.9000	1.0000	9/10	1	View Prompt View Tests
inspect	❌ FAIL	0.0000	0.0000	0/10	13	View Prompt View Tests
props	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests
snippets	❌ FAIL	0.0000	0.0000	0/10	10	View Prompt View Tests