SvelteBench Visualization - Legacy Results

Important: These are Legacy Results with Known Issues

These benchmark results contain an error in the "inspect" test prompt that may affect accuracy. The prompt had incorrect quotation marks in the Svelte binding syntax examples, which could confuse language models and lead to inconsistent results.

The updated results with corrected prompts are available separately.

GitHub View Current Results → v1 Raw Results

Top Models Leaderboard

Average pass@1 scores
Rank Model Score
1 claude-opus-4-1-20250805 (Anthropic)
88.9%
2 claude-opus-4-20250514 (Anthropic)
88.9%
3 claude-sonnet-4-20250514 (Anthropic)
88.9%
4 x-ai/grok-4 (OpenRouter)
87.8%
5 @preset/kimi-k2-0905-moonshotai (OpenRouter)
84.4%
6 moonshotai/kimi-k2 (OpenRouter)
84.4%
7 openrouter/sonoma-sky-alpha (OpenRouter)
84.4%
8 gemini-2.5-pro (Google)
84.4%
9 gemini-2.5-pro-preview-03-25 (Google)
83.3%
10 x-ai/grok-3-beta (OpenRouter)
83.3%
11 deepseek/deepseek-chat-v3.1 (OpenRouter)
82.2%
12 openrouter/sonoma-dusk-alpha (OpenRouter)
82.2%
13 gemini-2.5-pro-preview-06-05 (Google)
81.1%
14 gemini-2.5-pro-preview-05-06 (Google)
80.0%
15 glm-4.5-x (Z.ai)
80.0%
16 x-ai/grok-code-fast-1 (OpenRouter)
80.0%
17 gpt-5-2025-08-07 (OpenAI)
78.9%
18 glm-4.5 (Z.ai)
78.9%
19 gpt-5-2025-08-07-reasoning-medium (OpenAI)
77.8%
20 qwen/qwen3-max (OpenRouter)
74.4%
21 claude-3-5-haiku-20241022 (Anthropic)
73.3%
22 bytedance/seed-oss-36b-instruct (OpenRouter)
73.3%
23 mistralai/mistral-medium-3 (OpenRouter)
72.2%
24 gemini-2.5-flash (Google)
71.1%
25 gemini-2.5-flash-preview-04-17 (Google)
70.0%
26 qwen/qwen3-coder (OpenRouter)
67.8%
27 x-ai/grok-3-mini-beta (OpenRouter)
67.8%
28 z-ai/glm-4.5 (OpenRouter)
66.7%
29 meta-llama/llama-4-maverick (OpenRouter)
64.4%
30 mistralai/codestral-2508 (OpenRouter)
58.9%
31 glm-4.5-air (Z.ai)
58.9%
32 qwen/qwen3-235b-a22b-07-25 (OpenRouter)
57.8%
33 qwen/qwen3-235b-a22b-thinking-2507 (OpenRouter)
57.8%
34 z-ai/glm-4.5-air (OpenRouter)
57.8%
35 claude-3-7-sonnet-20250219 (Anthropic)
56.7%
36 glm-4.5-airx (Z.ai)
55.6%
37 mistralai/devstral-medium (OpenRouter)
52.2%
38 deepseek/deepseek-r1-0528 (OpenRouter)
48.9%
39 gemini-2.5-flash-lite (Google)
48.9%
40 z-ai/glm-4-32b (OpenRouter)
46.7%
41 glm-4-32b-0414-128k (Z.ai)
44.4%
42 mistralai/mistral-medium-3.1 (OpenRouter)
41.1%
43 openai/gpt-oss-120b (OpenRouter)
35.6%
44 qwen/qwen3-30b-a3b (OpenRouter)
34.4%
45 o3-2025-04-16 (OpenAI)
30.0%
46 chatgpt-4o-latest (OpenAI)
25.6%
47 gpt-4.1-2025-04-14 (OpenAI)
22.2%
48 gpt-5-mini-2025-08-07 (OpenAI)
21.1%
49 openai/gpt-oss-20b (OpenRouter)
20.0%
50 gpt-4o-2024-08-06 (OpenAI)
17.8%
51 gpt-5-nano-2025-08-07 (OpenAI)
16.7%
52 o3-mini-2025-01-31 (OpenAI)
15.6%
53 meta-llama/llama-4-scout (OpenRouter)
15.6%
54 o4-mini-2025-04-16 (OpenAI)
13.3%
55 mistralai/devstral-small (OpenRouter)
13.3%
56 gemma-3-27b-it (Google)
11.1%
57 gpt-4.1-nano-2025-04-14 (OpenAI)
11.1%
58 o1-pro-2025-03-19 (OpenAI)
11.1%
59 baidu/ernie-4.5-21b-a3b (OpenRouter)
11.1%
60 nousresearch/hermes-4-405b (OpenRouter)
11.1%
61 qwen/qwen3-30b-a3b-instruct-2507 (OpenRouter)
10.0%
62 ai21/jamba-large-1.7 (OpenRouter)
8.9%
63 ai21/jamba-mini-1.7 (OpenRouter)
8.9%
64 google/gemma-3n-e4b-it (OpenRouter)
8.9%
65 nousresearch/hermes-4-70b (OpenRouter)
6.7%
66 moonshotai/kimi-dev-72b:free (OpenRouter)
3.3%
67 gpt-4.1-mini-2025-04-14 (OpenAI)
1.1%

Note: Certain OpenAI thinking models (o3, o4) and gpt-5 do not support temperature adjustments (only default value of 1 is supported). Models with "-reasoning-" suffix (e.g., gpt-5-2025-08-07-reasoning-medium) will use the specified reasoning effort setting.

Errata: The "inspect" test has known correctness issues but is retained in the benchmark suite to maintain consistency and fairness in scoring across all evaluated models.

Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 60% 100% 6/10 4
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 10% 100% 1/10 9
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 40% 100% 4/10 6
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 20% 100% 2/10 8
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 3
Prompt Tests
derived 50% 100% 5/10 10
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 10% 100% 1/10 9
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 30% 100% 3/10 7
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 60% 100% 6/10 4
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 70% 100% 7/10 6
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 40% 100% 4/10 6
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 3
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 10% 100% 1/10 9
Prompt Tests
each 40% 100% 4/10 6
Prompt Tests
effect 0% 0% 0/10 13
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 3
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 20% 100% 2/10 9
Prompt Tests
effect 10% 100% 1/10 10
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 19
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 13
Prompt Tests
derived 0% 0% 0/10 13
Prompt Tests
derived-by 0% 0% 0/10 12
Prompt Tests
each 10% 100% 1/10 10
Prompt Tests
effect 0% 0% 0/10 16
Prompt Tests
hello-world 0% 0% 0/10 10
Prompt Tests
inspect 0% 0% 0/10 28
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 14
Prompt Tests
derived 0% 0% 0/10 13
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 13
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 13
Prompt Tests
derived 0% 0% 0/10 14
Prompt Tests
derived-by 40% 100% 4/10 14
Prompt Tests
each 0% 0% 0/10 11
Prompt Tests
effect 20% 100% 2/10 14
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 14
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 70% 100% 7/10 5
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 70% 100% 7/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 25
Prompt Tests
props 90% 100% 9/10 4
Prompt Tests
snippets 90% 100% 9/10 3
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 70% 100% 7/10 3
Prompt Tests
derived-by 60% 100% 6/10 6
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 70% 100% 7/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 40% 100% 4/10 6
Prompt Tests
derived 0% 0% 0/10 11
Prompt Tests
derived-by 0% 0% 0/10 12
Prompt Tests
each 50% 100% 5/10 5
Prompt Tests
effect 10% 100% 1/10 12
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 50% 100% 5/10 8
Prompt Tests
derived 0% 0% 0/10 13
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 13
Prompt Tests
snippets 0% 0% 0/10 24
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/1 1
Prompt Tests
derived 0% 0% 0/1 1
Prompt Tests
derived-by 0% 0% 0/1 3
Prompt Tests
each 0% 0% 0/1 1
Prompt Tests
effect 0% 0% 0/1 1
Prompt Tests
hello-world 100% 100% 1/1 0
Prompt Tests
inspect 0% 0% 0/1 1
Prompt Tests
props 0% 0% 0/1 1
Prompt Tests
snippets 0% 0% 0/1 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 60% 100% 6/10 4
Prompt Tests
derived 10% 100% 1/10 11
Prompt Tests
derived-by 20% 100% 2/10 8
Prompt Tests
each 60% 100% 6/10 4
Prompt Tests
effect 20% 100% 2/10 11
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 30% 100% 3/10 13
Prompt Tests
derived 0% 0% 0/10 11
Prompt Tests
derived-by 10% 100% 1/10 9
Prompt Tests
each 10% 100% 1/10 9
Prompt Tests
effect 0% 0% 0/10 17
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 10% 100% 1/10 12
Prompt Tests
derived 0% 0% 0/10 12
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 20% 100% 2/10 8
Prompt Tests
effect 0% 0% 0/10 11
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 19
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 90% 100% 9/10 2
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 11
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 13
Prompt Tests
derived 0% 0% 0/10 11
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 11
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 31
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 16
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 60% 100% 6/10 8
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 30% 100% 3/10 22
Prompt Tests
derived 60% 100% 6/10 6
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 20% 100% 2/10 10
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 40% 100% 4/10 6
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 19
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 1
Prompt Tests
derived 20% 100% 2/10 8
Prompt Tests
derived-by 0% 0% 0/10 11
Prompt Tests
each 0% 0% 0/10 11
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 30% 100% 3/10 11
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 60% 100% 6/10 4
Prompt Tests
each 70% 100% 7/10 4
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 22
Prompt Tests
props 10% 100% 1/10 9
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 80% 100% 8/10 4
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 10% 100% 1/10 9
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 19
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 6
Prompt Tests
derived 10% 100% 1/10 13
Prompt Tests
derived-by 10% 100% 1/10 13
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 30% 100% 3/10 7
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 80% 100% 8/10 2
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 80% 100% 8/10 2
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 50% 100% 5/10 5
Prompt Tests
each 20% 100% 2/10 8
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 0% 0% 0/10 10
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 30% 100% 3/10 8
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 22
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 70% 100% 7/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 70% 100% 7/10 3
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 10
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 13
Prompt Tests
hello-world 20% 100% 2/10 15
Prompt Tests
inspect 0% 0% 0/10 22
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 10% 100% 1/10 9
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 12
Prompt Tests
each 0% 0% 0/10 11
Prompt Tests
effect 0% 0% 0/10 13
Prompt Tests
hello-world 50% 100% 5/10 6
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 0% 0% 0/10 14
Prompt Tests
derived-by 30% 100% 3/10 7
Prompt Tests
each 50% 100% 5/10 6
Prompt Tests
effect 60% 100% 6/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 19
Prompt Tests
props 0% 0% 0/10 10
Prompt Tests
snippets 0% 0% 0/10 14
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 20% 100% 2/10 8
Prompt Tests
derived 0% 0% 0/10 15
Prompt Tests
derived-by 0% 0% 0/10 14
Prompt Tests
each 60% 100% 6/10 5
Prompt Tests
effect 0% 0% 0/10 16
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 16
Prompt Tests
snippets 0% 0% 0/10 12
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 50% 100% 5/10 5
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 19
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 2
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 40
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 90% 100% 9/10 1
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 90% 100% 9/10 1
Prompt Tests
derived-by 80% 100% 8/10 2
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 50% 100% 5/10 5
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 31
Prompt Tests
props 20% 100% 2/10 20
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 80% 100% 8/10 3
Prompt Tests
derived-by 60% 100% 6/10 4
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 3
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 20% 100% 2/10 11
Prompt Tests
snippets 0% 0% 0/10 14
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 60% 100% 6/10 7
Prompt Tests
derived 30% 100% 3/10 8
Prompt Tests
derived-by 30% 100% 3/10 9
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 30% 100% 3/10 11
Prompt Tests
hello-world 70% 100% 7/10 5
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 0% 0% 0/10 22
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 0% 0% 0/10 10
Prompt Tests
derived 0% 0% 0/10 10
Prompt Tests
derived-by 0% 0% 0/10 12
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 0% 0% 0/10 10
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 0% 0% 0/10 16
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 30% 100% 3/10 21
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 37
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 80% 100% 8/10 2
Prompt Tests
inspect 0% 0% 0/10 34
Prompt Tests
props 90% 100% 9/10 1
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 60% 100% 6/10 7
Prompt Tests
each 90% 100% 9/10 1
Prompt Tests
effect 70% 100% 7/10 6
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 40
Prompt Tests
props 80% 100% 8/10 2
Prompt Tests
snippets 10% 100% 1/10 11
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 90% 100% 9/10 3
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 100% 100% 10/10 0
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 70% 100% 7/10 5
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 60% 100% 6/10 4
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 1
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 80% 100% 8/10 4
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 20% 100% 2/10 12
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 25
Prompt Tests
props 30% 100% 3/10 7
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 80% 100% 8/10 2
Prompt Tests
derived 40% 100% 4/10 9
Prompt Tests
derived-by 80% 100% 8/10 2
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 80% 100% 8/10 4
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 25
Prompt Tests
props 80% 100% 8/10 5
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 90% 100% 9/10 3
Prompt Tests
derived 70% 100% 7/10 4
Prompt Tests
derived-by 90% 100% 9/10 3
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 40% 100% 4/10 7
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 13
Prompt Tests
props 30% 100% 3/10 7
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 100% 100% 10/10 0
Prompt Tests
derived-by 60% 100% 6/10 12
Prompt Tests
each 0% 0% 0/10 10
Prompt Tests
effect 20% 100% 2/10 12
Prompt Tests
hello-world 90% 100% 9/10 1
Prompt Tests
inspect 0% 0% 0/10 25
Prompt Tests
props 30% 100% 3/10 10
Prompt Tests
snippets 0% 0% 0/10 10
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 70% 100% 7/10 6
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 90% 100% 9/10 2
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 50% 100% 5/10 5
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 60% 100% 6/10 5
Prompt Tests
derived-by 80% 100% 8/10 2
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 60% 100% 6/10 4
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 20% 100% 2/10 8
Prompt Tests
snippets 10% 100% 1/10 9
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 40% 100% 4/10 8
Prompt Tests
derived-by 90% 100% 9/10 1
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 40% 100% 4/10 7
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 10
Prompt Tests
props 10% 100% 1/10 9
Prompt Tests
snippets 20% 100% 2/10 8
Prompt Tests
Test pass@1 pass@10 Passing Samples Errors Actions
counter 100% 100% 10/10 0
Prompt Tests
derived 50% 100% 5/10 10
Prompt Tests
derived-by 100% 100% 10/10 0
Prompt Tests
each 100% 100% 10/10 0
Prompt Tests
effect 100% 100% 10/10 0
Prompt Tests
hello-world 100% 100% 10/10 0
Prompt Tests
inspect 0% 0% 0/10 16
Prompt Tests
props 100% 100% 10/10 0
Prompt Tests
snippets 70% 100% 7/10 3
Prompt Tests