* Local models: Models in the local models tab were run through Ollama on an AMD Ryzen™ 5 7640HS processor (50W TDP) with Radeon™ 760M RDNA3 graphics. Speed is the generation rate measured on that hardware. The average response time is an average value based on one response sample per test.
| Test | pass@1 | pass@10 | Passing Samples | Errors | Actions |
|---|---|---|---|---|---|
| counter | 60% | 100% | 6/10 | 9 | |
| derived | 50% | 100% | 5/10 | 6 | |
| derived-by | 40% | 100% | 4/10 | 10 | |
| each | 30% | 100% | 3/10 | 11 | |
| effect | 20% | 100% | 2/10 | 13 | |
| hello-world | 90% | 100% | 9/10 | 2 | |
| inspect | 0% | 0% | 0/10 | 10 | |
| props | 30% | 100% | 3/10 | 13 | |
| snippets | 10% | 100% | 1/10 | 9 |