SvelteBench Visualization

GitHub View Raw Results → v1 Results

Top Models Leaderboard

Average pass@1 scores
Rank Model Score
1 claude-sonnet-4-20250514 (Anthropic)
90.0%
2 claude-sonnet-4-5 (Anthropic)
90.0%
3 claude-opus-4-1-20250805 (Anthropic)
88.9%
4 moonshotai/kimi-k2-0905 (OpenRouter)
88.9%
5 glm-4.5 (Z.ai)
86.7%
6 kimi-k2-turbo-preview (Moonshot AI)
85.6%
7 qwen/qwen3-max (OpenRouter)
85.6%
8 x-ai/grok-4 (OpenRouter)
84.4%
9 gemini-2.5-pro (Google)
83.3%
10 qwen/qwen3-coder-plus (OpenRouter)
83.3%
11 x-ai/grok-code-fast-1 (OpenRouter)
83.3%
12 openrouter/sonoma-sky-alpha (OpenRouter)
81.1%
13 gpt-5 (OpenAI)
81.1%
14 x-ai/grok-4-fast:free (OpenRouter)
80.0%
15 bytedance/seed-oss-36b-instruct (OpenRouter)
78.9%
16 @preset/alibaba-qwen3-vl-235b-a22b-instruct (OpenRouter)
77.8%
17 openrouter/sonoma-dusk-alpha (OpenRouter)
77.8%
18 deepseek/deepseek-v3.1-terminus (OpenRouter)
76.7%
19 gemini-2.5-flash (Google)
74.4%
20 qwen/qwen3-coder (OpenRouter)
73.3%
21 deepseek/deepseek-chat-v3.1 (OpenRouter)
70.0%
22 gemini-2.5-flash-lite-preview-09-2025 (Google)
67.8%
23 qwen/qwen3-235b-a22b-thinking-2507 (OpenRouter)
67.8%
24 gemini-2.5-flash-preview-09-2025 (Google)
66.7%
25 gemini-2.5-flash-lite (Google)
60.0%
26 qwen/qwen-plus-2025-07-28 (OpenRouter)
57.8%
27 gpt-5-codex (OpenAI)
57.8%
28 openai/gpt-5-codex (OpenRouter)
54.4%
29 meituan/longcat-flash-chat (OpenRouter)
52.2%
30 qwen/qwen3-next-80b-a3b-thinking (OpenRouter)
48.9%
31 glm-4-32b-0414-128k (Z.ai)
44.4%
32 qwen/qwen3-coder-flash (OpenRouter)
36.7%
33 qwen/qwen3-30b-a3b (OpenRouter)
26.7%
34 qwen/qwen3-next-80b-a3b-instruct (OpenRouter)
25.6%
35 o3 (OpenAI)
22.2%
36 gpt-5-mini (OpenAI)
20.0%
37 gpt-5-nano (OpenAI)
18.9%
38 o4-mini (OpenAI)
15.6%
39 opengvlab/internvl3-78b (OpenRouter)
15.6%
40 qwen/qwq-32b (OpenRouter)
14.4%
41 alibaba/tongyi-deepresearch-30b-a3b (OpenRouter)
12.2%
42 qwen/qwen-turbo (OpenRouter)
8.9%
43 arcee-ai/afm-4.5b (OpenRouter)
5.6%
44 nvidia/nemotron-nano-9b-v2 (OpenRouter)
0.0%

Note: Certain OpenAI thinking models (o3, o4) and gpt-5 do not support temperature adjustments (only default value of 1 is supported). Models with "-reasoning-" suffix (e.g., gpt-5-2025-08-07-reasoning-medium) will use the specified reasoning effort setting.

Errata: The "inspect" test has known correctness issues but is retained in the benchmark suite to maintain consistency and fairness in scoring across all evaluated models.

Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 20% 100% 2/10 8
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 9
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 9
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 80% 100% 8/10 4
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 10% 100% 1/10 9
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 70% 100% 7/10 3
Prompt Tests
props 20% 100% 2/10 8
Prompt Tests
snippets 0% 0% 0/10 14
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 3
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 20% 100% 2/10 8
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 80% 100% 8/10 2
Prompt Tests
each 30% 100% 3/10 7
Prompt Tests
effect 70% 100% 7/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 30% 100% 3/10 7
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 80% 100% 8/10 2
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 80% 100% 8/10 3
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 60% 100% 6/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 15
Prompt Tests
props 80% 100% 8/10 2
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 50% 100% 5/10 6
Prompt Tests
derived-by 50% 100% 5/10 5
Prompt Tests
each 70% 100% 7/10 3
Prompt Tests
effect 50% 100% 5/10 5
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 60% 100% 6/10 4
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 40% 100% 4/10 6
Prompt Tests
derived 0% 0% 0/10 11
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 30% 100% 3/10 8
Prompt Tests
effect 20% 100% 2/10 8
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 50% 100% 5/10 11
Prompt Tests
derived 10% 100% 1/10 11
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 12
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 9
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 18
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 10% 100% 1/10 9
Prompt Tests
derived 0% 0% 0/10 15
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 60% 100% 6/10 4
Prompt Tests
effect 0% 0% 0/10 12
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 9
Prompt Tests
props 20% 100% 2/10 8
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 10% 100% 1/10 9
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 10% 100% 1/10 10
Prompt Tests
effect 20% 100% 2/10 9
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 10% 100% 1/10 9
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 90% 100% 9/10 1
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 30% 100% 3/10 7
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 22
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 11
Prompt Tests
each 0% 0% 0/10 13
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 50% 100% 5/10 6
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 0% 0% 0/10 13
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 1
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 40% 100% 4/10 8
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 100% 100% 10/10 0
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 0% 0% 0/10 14
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 20% 100% 2/10 10
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 40% 100% 4/10 11
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 30% 100% 3/10 7
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 80% 100% 8/10 3
Prompt Tests
derived-by 90% 100% 9/10 3
Prompt Tests
each 40% 100% 4/10 12
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 30% 100% 3/10 7
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 70% 100% 7/10 3
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 20% 100% 2/10 8
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 10% 100% 1/10 9
Prompt Tests
each 10% 100% 1/10 9
Prompt Tests
effect 50% 100% 5/10 5
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 70% 100% 7/10 3
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 20% 100% 2/10 8
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 80% 100% 8/10 3
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 40% 100% 4/10 6
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 0% 0% 0/10 10
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 60% 100% 6/10 5
Prompt Tests
derived-by 20% 100% 2/10 8
Prompt Tests
each 70% 100% 7/10 3
Prompt Tests
effect 70% 100% 7/10 3
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 50% 100% 5/10 11
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 40% 100% 4/10 6
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 10% 100% 1/10 9
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 30% 100% 3/10 7
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 12
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 70% 100% 7/10 5
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 70% 100% 7/10 6
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 20% 100% 2/10 29
Prompt Tests
props 80% 100% 8/10 2
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 80% 100% 8/10 2
Prompt Tests
derived-by 70% 100% 7/10 5
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 50% 100% 5/10 5
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 40% 100% 4/10 6
Prompt Tests
props 10% 100% 1/10 12
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 10% 100% 1/10 10
Prompt Tests
hello-world 70% 100% 7/10 3
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 5
Prompt Tests
derived 60% 100% 6/10 5
Prompt Tests
derived-by 80% 100% 8/10 2
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 20% 100% 2/10 8
Prompt Tests
props 70% 100% 7/10 3
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 60% 100% 6/10 7
Prompt Tests
derived 20% 100% 2/10 8
Prompt Tests
derived-by 10% 100% 1/10 9
Prompt Tests
each 50% 100% 5/10 5
Prompt Tests
effect 0% 0% 0/10 11
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 16
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 30% 100% 3/10 15
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 90% 100% 9/10 1
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 50% 100% 5/10 17
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 40% 100% 4/10 6
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 80% 100% 8/10 3
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 10% 100% 1/10 9
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 70% 100% 7/10 5
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 80% 100% 8/10 5
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 100% 100% 10/10 0
Prompt Tests
props 70% 100% 7/10 3
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 50% 100% 5/10 5
Prompt Tests
derived 60% 100% 6/10 4
Prompt Tests
derived-by 10% 100% 1/10 13
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 14
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 10% 100% 1/10 9
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 40% 100% 4/10 18
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 60% 100% 6/10 9
Prompt Tests
each 80% 100% 8/10 2
Prompt Tests
effect 20% 100% 2/10 13
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 20% 100% 2/10 8
Prompt Tests
props 10% 100% 1/10 9
Prompt Tests
snippets 10% 100% 1/10 11
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 40% 100% 4/10 9
Prompt Tests
derived 0% 0% 0/10 13
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 11
Prompt Tests
effect 0% 0% 0/10 13
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 70% 100% 7/10 9
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 80% 100% 8/10 4
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 30% 100% 3/10 10
Prompt Tests
props 60% 100% 6/10 4
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 60% 100% 6/10 6
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 10% 100% 1/10 12
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 20% 100% 2/10 8
Prompt Tests
props 20% 100% 2/10 8
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 80% 100% 8/10 4
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 40% 100% 4/10 6
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests