SvelteBench Visualization

GitHub View Raw Results → v1 Results

Top Models Leaderboard

Average pass@1 scores
Rank Model Score
1 claude-opus-4-6 (Anthropic)
100.0%
2 claude-sonnet-4-6 (Anthropic)
100.0%
3 gemini-3.1-pro-preview (Google)
100.0%
4 anthropic/claude-opus-4.5 (OpenRouter)
100.0%
5 gpt-5.2 (OpenAI)
97.8%
6 gpt-5.3-codex (OpenAI)
96.7%
7 gpt-5.4 (OpenAI)
95.6%
8 minimax/minimax-m2.5 (OpenRouter)
95.6%
9 gemini-3-flash-preview (Google)
94.4%
10 gpt-5.3-chat-latest (OpenAI)
94.4%
11 openrouter/pony-alpha (OpenRouter)
94.4%
12 claude-sonnet-4-5 (Anthropic)
93.3%
13 openrouter/healer-alpha (OpenRouter)
93.3%
14 xiaomi/mimo-v2-flash:free (OpenRouter)
93.3%
15 kimi-k2-thinking-turbo (Moonshot)
92.2%
16 openrouter/sherlock-dash-alpha (OpenRouter)
92.2%
17 qwen/qwen3.5-27b (OpenRouter)
92.2%
18 deepseek/deepseek-v3.2 (OpenRouter)
91.1%
19 openrouter/hunter-alpha (OpenRouter)
91.1%
20 qwen/qwen3.5-122b-a10b (OpenRouter)
91.1%
21 moonshotai/kimi-k2-0905 (OpenRouter)
90.0%
22 claude-sonnet-4-20250514 (Anthropic)
90.0%
23 gemini-3-pro-preview (Google)
90.0%
24 moonshotai/kimi-k2.5 (OpenRouter)
90.0%
25 kimi-k2-thinking (Moonshot)
88.9%
26 gpt-5-chat-latest (OpenAI)
88.9%
27 kwaipilot/kat-coder-pro:free (OpenRouter)
88.9%
28 minimax/minimax-m2.1 (OpenRouter)
88.9%
29 mistralai/mistral-large-2512 (OpenRouter)
88.9%
30 qwen/qwen3-max (OpenRouter)
88.9%
31 qwen/qwen3.5-397b-a17b (OpenRouter)
88.9%
32 qwen/qwen3.5-plus-02-15 (OpenRouter)
88.9%
33 grok-4.20-experimental-beta-0304-non-reasoning (xAI)
88.9%
34 grok-code-fast-1 (xAI)
88.9%
35 glm-5 (Z.ai)
88.9%
36 gpt-5.2-codex (OpenAI)
87.8%
37 deepseek/deepseek-v3.2-speciale (OpenRouter)
87.8%
38 z-ai/glm-4.7 (OpenRouter)
87.8%
39 grok-4.20-experimental-beta-0304-reasoning (xAI)
87.8%
40 deepseek/deepseek-v3.2-exp (OpenRouter)
86.7%
41 gemini-2.5-pro (Google)
85.6%
42 gpt-5 (OpenAI)
85.6%
43 z-ai/glm-4.6 (OpenRouter)
84.4%
44 claude-haiku-4-5-20251001 (Anthropic)
84.4%
45 gemini-3.1-flash-lite-preview (Google)
84.4%
46 gpt-5.1-chat-latest (OpenAI)
83.3%
47 openai/gpt-5.1-chat (OpenRouter)
83.3%
48 bytedance-seed/seed-2.0-lite (OpenRouter)
80.0%
49 openai/gpt-5.1 (OpenRouter)
80.0%
50 openrouter/sherlock-think-alpha (OpenRouter)
80.0%
51 stepfun/step-3.5-flash:free (OpenRouter)
80.0%
52 x-ai/grok-4.1-fast (OpenRouter)
80.0%
53 qwen/qwen3.5-flash-02-23 (OpenRouter)
80.0%
54 bytedance-seed/seed-2.0-mini (OpenRouter)
78.9%
55 qwen/qwen3-max-thinking (OpenRouter)
78.9%
56 qwen/qwen3.5-35b-a3b (OpenRouter)
77.8%
57 bytedance-seed/seed-1.6 (OpenRouter)
76.7%
58 minimax/minimax-m2 (OpenRouter)
75.6%
59 openrouter/polaris-alpha (OpenRouter)
73.3%
60 qwen/qwen3-coder-next (OpenRouter)
71.1%
61 nvidia/nemotron-3-super-120b-a12b:free (OpenRouter)
70.0%
62 gpt-5.1-codex-max (OpenAI)
67.8%
63 moonshotai/kimi-linear-48b-a3b-instruct (OpenRouter)
66.7%
64 z-ai/glm-4.7-flash (OpenRouter)
60.0%
65 amazon/nova-2-lite-v1:free (OpenRouter)
56.7%
66 prime-intellect/intellect-3 (OpenRouter)
55.6%
67 mistralai/devstral-2512:free (OpenRouter)
50.0%
68 gpt-5-codex (OpenAI)
47.8%
69 openai/gpt-5.1-codex (OpenRouter)
45.6%
70 inception/mercury-2 (OpenRouter)
44.4%
71 qwen/qwen3.5-9b (OpenRouter)
42.2%
72 bytedance-seed/seed-1.6-flash (OpenRouter)
35.6%
73 openrouter/aurora-alpha (OpenRouter)
34.4%
74 essentialai/rnj-1-instruct (OpenRouter)
30.0%
75 gpt-5-mini (OpenAI)
23.3%
76 arcee-ai/trinity-large-preview:free (OpenRouter)
23.3%
77 mistralai/ministral-3b-2512 (OpenRouter)
22.2%
78 upstage/solar-pro-3:free (OpenRouter)
20.0%
79 nvidia/nemotron-3-nano-30b-a3b:free (OpenRouter)
18.9%
80 openai/gpt-5.1-codex-mini (OpenRouter)
16.7%
81 gpt-5-nano (OpenAI)
13.3%
82 liquid/lfm-2-24b-a2b (OpenRouter)
13.3%
83 qwen/qwen3-vl-8b-instruct (OpenRouter)
11.1%
84 deepcogito/cogito-v2-preview-llama-405b (OpenRouter)
10.0%
85 mistralai/ministral-8b-2512 (OpenRouter)
6.7%
86 mistralai/ministral-14b-2512 (OpenRouter)
3.3%
87 allenai/olmo-3.1-32b-instruct (OpenRouter)
2.2%
88 allenai/olmo-3.1-32b-think:free (OpenRouter)
0.0%

Note: Certain OpenAI thinking models (o3, o4) and gpt-5 do not support temperature adjustments (only default value of 1 is supported). Models with "-reasoning-" suffix (e.g., gpt-5-2025-08-07-reasoning-medium) will use the specified reasoning effort setting.

Errata: The "inspect" test has known correctness issues but is retained in the benchmark suite to maintain consistency and fairness in scoring across all evaluated models.

Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 60% 100% 6/10 16
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 100% 100% 10/10 0
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 20% 100% 2/10 8
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 40% 100% 4/10 6
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 100% 100% 10/10 0
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 70% 100% 7/10 3
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 60% 100% 6/10 4
Prompt Tests
props 100% 100% 9/9 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 9
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 11
Prompt Tests
props 70% 100% 7/10 3
Prompt Tests
snippets 80% 100% 8/10 2
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 100% 100% 10/10 0
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 9
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 30% 100% 3/10 7
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 18
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 100% 100% 10/10 0
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 60% 100% 6/10 10
Prompt Tests
derived 40% 100% 4/10 6
Prompt Tests
derived-by 30% 100% 3/10 7
Prompt Tests
each 70% 100% 7/10 3
Prompt Tests
effect 20% 100% 2/10 9
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 40% 100% 4/10 6
Prompt Tests
snippets 70% 100% 7/10 3
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 50% 100% 5/10 5
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 50% 100% 5/10 5
Prompt Tests
effect 0% 0% 0/10 11
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 20% 100% 2/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 20% 100% 2/10 8
Prompt Tests
derived 0% 0% 0/10 13
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 11
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 12
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 18
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 70% 100% 7/10 6
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 80% 100% 8/10 5
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 20% 100% 2/10 8
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 70% 100% 7/10 3
Prompt Tests
derived-by 40% 100% 4/10 8
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 60% 100% 6/10 4
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 2
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 90% 100% 9/10 1
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 30% 100% 3/10 13
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 60% 100% 6/10 4
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 2
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 70% 100% 7/10 8
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 70% 100% 7/10 9
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 60% 100% 6/10 12
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 10% 100% 1/10 11
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 10% 100% 1/10 9
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 0% 0% 0/10 10
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 3
Prompt Tests
derived 80% 100% 8/10 2
Prompt Tests
derived-by 70% 100% 7/10 3
Prompt Tests
each 70% 100% 7/10 4
Prompt Tests
effect 60% 100% 6/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 30% 100% 3/10 7
Prompt Tests
props 30% 100% 3/10 7
Prompt Tests
snippets 0% 0% 0/10 12
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 100% 100% 10/10 0
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 60% 100% 6/10 7
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 11
Prompt Tests
effect 60% 100% 6/10 4
Prompt Tests
hello-world 80% 100% 8/10 4
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 10% 100% 1/10 15
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 3
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 80% 100% 8/10 2
Prompt Tests
props 20% 100% 2/10 8
Prompt Tests
snippets 10% 100% 1/10 11
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 3
Prompt Tests
derived 60% 100% 6/10 4
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 30% 100% 3/10 7
Prompt Tests
effect 0% 0% 0/10 16
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 50% 100% 5/10 5
Prompt Tests
props 10% 100% 1/10 9
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 30% 100% 3/10 7
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 30% 100% 3/10 7
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 11
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 12
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 30% 100% 3/10 7
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 1
Prompt Tests
derived 80% 100% 8/10 2
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 20% 100% 2/10 8
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 3
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 20% 100% 2/10 8
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 70% 100% 7/10 3
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 40% 100% 4/10 9
Prompt Tests
derived 50% 100% 5/10 7
Prompt Tests
derived-by 40% 100% 4/10 8
Prompt Tests
each 20% 100% 2/10 8
Prompt Tests
effect 10% 100% 1/10 10
Prompt Tests
hello-world 60% 100% 6/10 6
Prompt Tests
inspect 30% 100% 3/10 7
Prompt Tests
props 0% 0% 0/10 13
Prompt Tests
snippets 20% 100% 2/10 14
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 0% 0% 0/10 15
Prompt Tests
derived-by 50% 100% 5/10 9
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 50% 100% 5/10 6
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 40% 100% 4/10 6
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 90% 100% 9/10 1
Prompt Tests
props 70% 100% 7/10 3
Prompt Tests
snippets 40% 100% 4/10 8
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 10% 100% 1/10 25
Prompt Tests
derived 20% 100% 2/10 11
Prompt Tests
derived-by 0% 0% 0/10 24
Prompt Tests
each 0% 0% 0/10 11
Prompt Tests
effect 0% 0% 0/10 12
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 16
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 8
Prompt Tests
derived 80% 100% 8/10 4
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 80% 100% 8/10 3
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 9/9 0
Prompt Tests
inspect 30% 100% 3/10 10
Prompt Tests
props 70% 100% 7/10 6
Prompt Tests
snippets 40% 100% 4/10 8
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 30% 100% 3/10 10
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 80% 100% 8/10 2
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 60% 100% 6/10 4
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 40% 100% 4/10 6
Prompt Tests
inspect 10% 100% 1/10 9
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 30% 100% 3/10 7
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 0% 0% 0/10 10
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 12
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 14
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 11
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 15
Prompt Tests
props 0% 0% 0/10 22
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 60% 100% 6/10 5
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 100% 100% 10/10 0
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 40% 100% 4/10 6
Prompt Tests
props 80% 100% 8/10 3
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 9
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 70% 100% 7/10 3
Prompt Tests
each 30% 100% 3/10 7
Prompt Tests
effect 50% 100% 5/10 6
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 60% 100% 6/10 4
Prompt Tests
props 60% 100% 6/10 5
Prompt Tests
snippets 40% 100% 4/10 7
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 60% 100% 6/10 4
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 20
Prompt Tests
each 20% 100% 2/10 8
Prompt Tests
effect 0% 0% 0/10 20
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 10% 100% 1/10 20
Prompt Tests
props 0% 0% 0/10 11
Prompt Tests
snippets 0% 0% 0/10 22
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 4
Prompt Tests
derived 90% 100% 9/10 2
Prompt Tests
derived-by 70% 100% 7/10 7
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 50% 100% 5/10 19
Prompt Tests
props 30% 100% 3/10 7
Prompt Tests
snippets 10% 100% 1/10 11
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 80% 100% 8/10 2
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 30% 100% 3/10 15
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 40% 100% 4/10 6
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 1
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 80% 100% 8/10 4
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 80% 100% 8/10 5
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 30% 100% 3/10 7
Prompt Tests
derived-by 10% 100% 1/10 9
Prompt Tests
each 70% 100% 7/10 3
Prompt Tests
effect 20% 100% 2/10 8
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 50% 100% 5/10 10
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 30% 100% 3/10 7
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 14
Prompt Tests
each 50% 100% 5/10 6
Prompt Tests
effect 20% 100% 2/10 12
Prompt Tests
hello-world 50% 100% 5/10 5
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 0% 0% 0/10 16
Prompt Tests
derived-by 30% 100% 3/10 9
Prompt Tests
each 60% 100% 6/10 4
Prompt Tests
effect 20% 100% 2/10 9
Prompt Tests
hello-world 60% 100% 6/10 4
Prompt Tests
inspect 40% 100% 4/10 6
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 20% 100% 2/10 8
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 60% 100% 6/10 6
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 80% 100% 8/10 6
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 30% 100% 3/10 7
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 80% 100% 8/10 5
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 70% 100% 7/10 3
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 80% 100% 8/10 2
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 80% 100% 8/10 8
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 50% 100% 5/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 70% 100% 7/10 7
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 34
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 6
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 40% 100% 4/10 10
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 20% 100% 2/10 13
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 27
Prompt Tests
props 50% 100% 5/10 5
Prompt Tests
snippets 20% 100% 2/10 14
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 10% 100% 1/10 9
Prompt Tests
effect 70% 100% 7/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 60% 100% 6/10 9
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 20% 100% 2/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 9/9 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 90% 100% 9/10 1
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 19
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 3
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 90% 100% 9/10 1
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 80% 100% 8/10 2
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 70% 100% 7/10 3
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 60% 100% 6/10 6
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 70% 100% 7/10 6
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 50% 100% 5/10 14
Prompt Tests
props 80% 100% 8/10 2
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 8
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 60% 100% 6/10 4
Prompt Tests
inspect 70% 100% 7/10 3
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 60% 100% 6/10 7
Prompt Tests
derived 50% 100% 5/10 6
Prompt Tests
derived-by 60% 100% 6/10 9
Prompt Tests
each 50% 100% 5/10 9
Prompt Tests
effect 40% 100% 4/10 8
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 10% 100% 1/10 15
Prompt Tests
props 30% 100% 3/10 13
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 1
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 3
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 70% 100% 7/10 5
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 24
Prompt Tests
props 80% 100% 8/10 2
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 40% 100% 4/10 8
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 60% 100% 6/10 6
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 3
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 50% 100% 5/10 9
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 80% 100% 8/10 6
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 16
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 40% 100% 4/10 7
Prompt Tests
derived 0% 0% 0/10 12
Prompt Tests
derived-by 40% 100% 4/10 6
Prompt Tests
each 20% 100% 2/10 8
Prompt Tests
effect 0% 0% 0/10 11
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 12
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 60% 100% 6/10 7
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 31
Prompt Tests
props 90% 100% 9/10 2
Prompt Tests
snippets 70% 100% 7/10 3
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 70% 100% 7/10 3
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 70% 100% 7/10 7
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 40% 100% 4/10 12
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 30% 100% 3/10 7
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 9
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 80% 100% 8/10 2
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 1
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 50% 100% 5/10 5
Prompt Tests
each 60% 100% 6/10 4
Prompt Tests
effect 40% 100% 4/10 10
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 50% 100% 5/10 10
Prompt Tests
props 40% 100% 4/10 10
Prompt Tests
snippets 10% 100% 1/10 13
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 100% 100% 10/10 0
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 16
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 10% 100% 1/10 9
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 80% 100% 8/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 30% 100% 3/10 10
Prompt Tests
props 90% 100% 9/10 2
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 20% 100% 2/10 8
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 80% 100% 8/10 2
Prompt Tests