Local model eval suite
My own test bench for AI models. It gives local and cloud models the same coding tasks through six different agent tools and scores the results. It picked my everyday local model from 220 scored runs.
- Why
- Leaderboards didn't tell me which local model would actually do good work on my laptop, inside the tools I use. I wanted my own answer, measured on my own tasks.
- How long
- July to August 2026, then re-run when a new model lands
How I built it
- I wrote down the question first: which local model and agent tool should be my daily driver?
- I had the AI write coding tasks with precise, unambiguous instructions, because that's how a frontier model directs my local model in real use.
- I ran every model through six agent tools on the same tasks, with a cloud model as a reference bar.
- Every result goes into a log that is never edited, only added to.
- After 220 scored runs, I picked qwen3.6-35b-a3b with Pi, and now I re-run it whenever a new model lands.
- To copy it you need
- Ollama, a few local models, and Claude Code, or the AI of your choice, to write the tasks and the scoring. Gotcha: keep the tasks the same for every tool. If you tweak a prompt for one tool, the comparison is ruined.
Try it yourself
Copy this prompt and paste it into ChatGPT or Claude to try it yourself.
I want to choose which AI model to run on my own computer based on my own tasks, not a leaderboard. Korab Eland built a personal eval suite for this (on whatarewebuilding.ai): the same coding tasks go through several agent tools and local models, every run is scored and logged, and after 220 runs Korab picked a daily driver. Help me build a small version. First ask what I want the model to do and what computer I have. Then write five realistic tasks with precise instructions and a clear pass or fail check for each, pick three models to try, and keep a results log we only ever add to. Keep the tasks identical for every model. Ask me for anything you need.
Tools I used for this project
