SvelteBench Visualization

GitHub View Raw Results →

Top Models Leaderboard

Average pass@1 scores
Rank Model Score
1 claude-opus-4-1-20250805 (Anthropic)
88.9%
2 claude-opus-4-20250514 (Anthropic)
88.9%
3 claude-sonnet-4-20250514 (Anthropic)
88.9%
4 x-ai/grok-4 (OpenRouter)
87.8%
5 @preset/kimi-k2-0905-moonshotai (OpenRouter)
84.4%
6 moonshotai/kimi-k2 (OpenRouter)
84.4%
7 openrouter/sonoma-sky-alpha (OpenRouter)
84.4%
8 gemini-2.5-pro (Google)
84.4%
9 gemini-2.5-pro-preview-03-25 (Google)
83.3%
10 x-ai/grok-3-beta (OpenRouter)
83.3%
11 deepseek/deepseek-chat-v3.1 (OpenRouter)
82.2%
12 openrouter/sonoma-dusk-alpha (OpenRouter)
82.2%
13 gemini-2.5-pro-preview-06-05 (Google)
81.1%
14 x-ai/grok-3 (OpenRouter)
81.1%
15 gemini-2.5-pro-preview-05-06 (Google)
80.0%
16 glm-4.5-x (Z.ai)
80.0%
17 x-ai/grok-code-fast-1 (OpenRouter)
80.0%
18 gpt-5-2025-08-07 (OpenAI)
78.9%
19 glm-4.5 (Z.ai)
78.9%
20 gpt-5-2025-08-07-reasoning-medium (OpenAI)
77.8%
21 openrouter/horizon-alpha (OpenRouter)
77.8%
22 openrouter/horizon-beta (OpenRouter)
76.7%
23 qwen/qwen3-max (OpenRouter)
74.4%
24 claude-3-5-haiku-20241022 (Anthropic)
73.3%
25 bytedance/seed-oss-36b-instruct (OpenRouter)
73.3%
26 mistralai/mistral-medium-3 (OpenRouter)
72.2%
27 gemini-2.5-flash (Google)
71.1%
28 gemini-2.5-flash-preview-04-17 (Google)
70.0%
29 qwen/qwen3-coder (OpenRouter)
67.8%
30 x-ai/grok-3-mini-beta (OpenRouter)
67.8%
31 z-ai/glm-4.5 (OpenRouter)
66.7%
32 x-ai/grok-3-mini (OpenRouter)
64.4%
33 meta-llama/llama-4-maverick (OpenRouter)
64.4%
34 mistralai/codestral-2508 (OpenRouter)
58.9%
35 glm-4.5-air (Z.ai)
58.9%
36 qwen/qwen3-235b-a22b-07-25 (OpenRouter)
57.8%
37 qwen/qwen3-235b-a22b-thinking-2507 (OpenRouter)
57.8%
38 z-ai/glm-4.5-air (OpenRouter)
57.8%
39 claude-3-7-sonnet-20250219 (Anthropic)
56.7%
40 glm-4.5-airx (Z.ai)
55.6%
41 mistralai/devstral-medium (OpenRouter)
52.2%
42 deepseek/deepseek-r1-0528 (OpenRouter)
48.9%
43 gemini-2.5-flash-lite (Google)
48.9%
44 z-ai/glm-4-32b (OpenRouter)
46.7%
45 glm-4-32b-0414-128k (Z.ai)
44.4%
46 mistralai/mistral-medium-3.1 (OpenRouter)
41.1%
47 openai/gpt-oss-120b (OpenRouter)
35.6%
48 qwen/qwen3-30b-a3b (OpenRouter)
34.4%
49 o3-2025-04-16 (OpenAI)
30.0%
50 chatgpt-4o-latest (OpenAI)
25.6%
51 gpt-4.1-2025-04-14 (OpenAI)
22.2%
52 gpt-5-mini-2025-08-07 (OpenAI)
21.1%
53 openai/gpt-oss-20b (OpenRouter)
20.0%
54 gpt-4o-2024-08-06 (OpenAI)
17.8%
55 gpt-5-nano-2025-08-07 (OpenAI)
16.7%
56 o3-mini-2025-01-31 (OpenAI)
15.6%
57 meta-llama/llama-4-scout (OpenRouter)
15.6%
58 o4-mini-2025-04-16 (OpenAI)
13.3%
59 mistralai/devstral-small (OpenRouter)
13.3%
60 gemma-3-27b-it (Google)
11.1%
61 gpt-4.1-nano-2025-04-14 (OpenAI)
11.1%
62 o1-pro-2025-03-19 (OpenAI)
11.1%
63 baidu/ernie-4.5-21b-a3b (OpenRouter)
11.1%
64 nousresearch/hermes-4-405b (OpenRouter)
11.1%
65 qwen/qwen3-30b-a3b-instruct-2507 (OpenRouter)
10.0%
66 ai21/jamba-large-1.7 (OpenRouter)
8.9%
67 ai21/jamba-mini-1.7 (OpenRouter)
8.9%
68 google/gemma-3n-e4b-it (OpenRouter)
8.9%
69 nousresearch/hermes-4-70b (OpenRouter)
6.7%
70 moonshotai/kimi-dev-72b:free (OpenRouter)
3.3%
71 gpt-4.1-mini-2025-04-14 (OpenAI)
1.1%

Note: Certain OpenAI thinking models (o3, o4) and gpt-5 do not support temperature adjustments (only default value of 1 is supported). Models with "-reasoning-" suffix (e.g., gpt-5-2025-08-07-reasoning-medium) will use the specified reasoning effort setting.

Errata: The "inspect" test has known correctness issues but is retained in the benchmark suite to maintain consistency and fairness in scoring across all evaluated models.

Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 60% 100% 6/10 4
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 10% 100% 1/10 9
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 40% 100% 4/10 6
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 20% 100% 2/10 8
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 3
Prompt Tests
derived 50% 100% 5/10 10
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 10% 100% 1/10 9
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 30% 100% 3/10 7
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 60% 100% 6/10 4
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 70% 100% 7/10 6
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 40% 100% 4/10 6
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 3
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 10% 100% 1/10 9
Prompt Tests
each 40% 100% 4/10 6
Prompt Tests
effect 0% 0% 0/10 13
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 3
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 20% 100% 2/10 9
Prompt Tests
effect 10% 100% 1/10 10
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 19
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 13
Prompt Tests
derived 0% 0% 0/10 13
Prompt Tests
derived-by 0% 0% 0/10 12
Prompt Tests
each 10% 100% 1/10 10
Prompt Tests
effect 0% 0% 0/10 16
Prompt Tests
hello-world 0% 0% 0/10 10
Prompt Tests
inspect 0% 0% 0/10 28
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 14
Prompt Tests
derived 0% 0% 0/10 13
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 13
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 13
Prompt Tests
derived 0% 0% 0/10 14
Prompt Tests
derived-by 40% 100% 4/10 14
Prompt Tests
each 0% 0% 0/10 11
Prompt Tests
effect 20% 100% 2/10 14
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 14
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 70% 100% 7/10 5
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 70% 100% 7/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 25
Prompt Tests
props 90% 100% 9/10 4
Prompt Tests
snippets 90% 100% 9/10 3
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 70% 100% 7/10 3
Prompt Tests
derived-by 60% 100% 6/10 6
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 70% 100% 7/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 40% 100% 4/10 6
Prompt Tests
derived 0% 0% 0/10 11
Prompt Tests
derived-by 0% 0% 0/10 12
Prompt Tests
each 50% 100% 5/10 5
Prompt Tests
effect 10% 100% 1/10 12
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 50% 100% 5/10 8
Prompt Tests
derived 0% 0% 0/10 13
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 13
Prompt Tests
snippets 0% 0% 0/10 24
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/1 1
Prompt Tests
derived 0% 0% 0/1 1
Prompt Tests
derived-by 0% 0% 0/1 3
Prompt Tests
each 0% 0% 0/1 1
Prompt Tests
effect 0% 0% 0/1 1
Prompt Tests
hello-world 100% 100% 1/1 0
Prompt Tests
inspect 0% 0% 0/1 1
Prompt Tests
props 0% 0% 0/1 1
Prompt Tests
snippets 0% 0% 0/1 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 60% 100% 6/10 4
Prompt Tests
derived 10% 100% 1/10 11
Prompt Tests
derived-by 20% 100% 2/10 8
Prompt Tests
each 60% 100% 6/10 4
Prompt Tests
effect 20% 100% 2/10 11
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 30% 100% 3/10 13
Prompt Tests
derived 0% 0% 0/10 11
Prompt Tests
derived-by 10% 100% 1/10 9
Prompt Tests
each 10% 100% 1/10 9
Prompt Tests
effect 0% 0% 0/10 17
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 10% 100% 1/10 12
Prompt Tests
derived 0% 0% 0/10 12
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 20% 100% 2/10 8
Prompt Tests
effect 0% 0% 0/10 11
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 19
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 90% 100% 9/10 2
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 11
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 13
Prompt Tests
derived 0% 0% 0/10 11
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 11
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 31
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 16
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 60% 100% 6/10 8
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 30% 100% 3/10 22
Prompt Tests
derived 60% 100% 6/10 6
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 20% 100% 2/10 10
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 40% 100% 4/10 6
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 19
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 1
Prompt Tests
derived 20% 100% 2/10 8
Prompt Tests
derived-by 0% 0% 0/10 11
Prompt Tests
each 0% 0% 0/10 11
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 30% 100% 3/10 11
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 60% 100% 6/10 4
Prompt Tests
each 70% 100% 7/10 4
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 22
Prompt Tests
props 10% 100% 1/10 9
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 80% 100% 8/10 4
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 10% 100% 1/10 9
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 19
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 6
Prompt Tests
derived 10% 100% 1/10 13
Prompt Tests
derived-by 10% 100% 1/10 13
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 30% 100% 3/10 7
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 80% 100% 8/10 2
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 80% 100% 8/10 2
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 50% 100% 5/10 5
Prompt Tests
each 20% 100% 2/10 8
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 0% 0% 0/10 10
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 30% 100% 3/10 8
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 22
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 70% 100% 7/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 3
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 13
Prompt Tests
hello-world 20% 100% 2/10 15
Prompt Tests
inspect 0% 0% 0/10 22
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 10% 100% 1/10 9
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 12
Prompt Tests
each 0% 0% 0/10 11
Prompt Tests
effect 0% 0% 0/10 13
Prompt Tests
hello-world 50% 100% 5/10 6
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 0% 0% 0/10 14
Prompt Tests
derived-by 30% 100% 3/10 7
Prompt Tests
each 50% 100% 5/10 6
Prompt Tests
effect 60% 100% 6/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 19
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 14
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 20% 100% 2/10 8
Prompt Tests
derived 0% 0% 0/10 15
Prompt Tests
derived-by 0% 0% 0/10 14
Prompt Tests
each 60% 100% 6/10 5
Prompt Tests
effect 0% 0% 0/10 16
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 16
Prompt Tests
snippets 0% 0% 0/10 12
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 22
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 25
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 50% 100% 5/10 5
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 19
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 2
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 40
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 80% 100% 8/10 2
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 50% 100% 5/10 5
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 31
Prompt Tests
props 20% 100% 2/10 20
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 80% 100% 8/10 3
Prompt Tests
derived-by 60% 100% 6/10 4
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 20% 100% 2/10 11
Prompt Tests
snippets 0% 0% 0/10 14
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 60% 100% 6/10 7
Prompt Tests
derived 30% 100% 3/10 8
Prompt Tests
derived-by 30% 100% 3/10 9
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 30% 100% 3/10 11
Prompt Tests
hello-world 70% 100% 7/10 5
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 22
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 12
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 16
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 30% 100% 3/10 21
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 37
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 34
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 80% 100% 8/10 2
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 50% 100% 5/10 9
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 50% 100% 5/10 10
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 31
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 60% 100% 6/10 7
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 70% 100% 7/10 6
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 40
Prompt Tests
props 80% 100% 8/10 2
Prompt Tests
snippets 10% 100% 1/10 11
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 3
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 70% 100% 7/10 5
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 1
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 80% 100% 8/10 4
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 20% 100% 2/10 12
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 25
Prompt Tests
props 30% 100% 3/10 7
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 40% 100% 4/10 9
Prompt Tests
derived-by 80% 100% 8/10 2
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 25
Prompt Tests
props 80% 100% 8/10 5
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 3
Prompt Tests
derived 70% 100% 7/10 4
Prompt Tests
derived-by 90% 100% 9/10 3
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 40% 100% 4/10 7
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 30% 100% 3/10 7
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 60% 100% 6/10 12
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 20% 100% 2/10 12
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 25
Prompt Tests
props 30% 100% 3/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 70% 100% 7/10 6
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 60% 100% 6/10 5
Prompt Tests
derived-by 80% 100% 8/10 2
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 60% 100% 6/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 20% 100% 2/10 8
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 40% 100% 4/10 8
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 40% 100% 4/10 7
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 10% 100% 1/10 9
Prompt Tests
snippets 20% 100% 2/10 8
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 50% 100% 5/10 10
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 70% 100% 7/10 3
Prompt Tests