# Netlify tested 11 coding models side by side

> Netlify ran the same build prompt across 11 AI models using their open-source AXIS evaluator. Here is what the results tell us about model selection for code generation.

- Published: 2026-08-14
- Canonical: https://aicodereview.io/blog/netlify-tested-11-coding-models-side-by-side/
- Author: aicodereview.io Editorial

---
Netlify published an experiment this week that more teams should run. They tested 11 different AI models on the same three coding prompts, using their open-source AXIS evaluation tool to score the results. Same task, same agent framework, same evaluation criteria. Only the model changed.

The test covered three scenarios: a static coffee-shop site, a to-do list app with a database, and a recipe app that calls an AI inference API. Each model ran three times per prompt. The results are published at the-coffee-shop-brief.netlify.app for anyone to inspect.

Some things that stood out.

**Cost variance was wide**

On the simple coffee-shop site, average credit cost ranged from 103 (Gemini 3.6 Flash) to 519 (Claude Opus 5). That is a 5x difference for a static one-pager. The gap would shrink on harder tasks where cheaper models fail more and require retries, but for straightforward work the cost spread is real.

**Output quality was not uniform**

While Netlify focused on functional correctness rather than aesthetics, the generated sites differed meaningfully. Some models picked a reasonable color palette and layout; others produced broken navigation or misused database primitives. The full report includes links to each generated site so you can judge visually.

**Structured evaluation beats vibes**

Netlify used their AXIS framework, which defines pass/fail checks programmatically (does the site use a database when needed, does it call the right API, is the site over-engineered). This catches regressions that manual review would miss. AXIS is open-source, so teams can adapt it to their own standards.

The practical takeaway: model selection for code generation should be an empirical choice, not a brand preference. Run your prompts on 3-5 models. Measure pass rates and cost. The results will surprise you.

Netlify's post hinted at follow-ups covering the harder scenarios. I will run the same methodology on my own test suite and report back with numbers.

## FAQ

### What did Netlify test in its AXIS model comparison?

Netlify ran 11 AI models on the same three coding prompts — a static coffee-shop site, a to-do list app with a database, and a recipe app that calls an AI inference API — each run three times per prompt, scored with their open-source AXIS evaluation tool.

### How much did AI model cost vary for the same coding task in Netlify's test?

On the simple coffee-shop site, average credit cost ranged from 103 (Gemini 3.6 Flash) to 519 (Claude Opus 5) — a 5x difference for a static one-pager.

### What does Netlify's AXIS evaluation framework actually check?

AXIS defines pass/fail checks programmatically, such as whether the site uses a database when needed, whether it calls the right API, and whether the output is over-engineered, catching regressions that manual review would miss.

### Where can you see the actual sites generated in Netlify's model comparison?

The full results, including links to each generated site, are published at the-coffee-shop-brief.netlify.app.

### What's the practical takeaway from Netlify's multi-model coding test?

Model selection for code generation should be an empirical choice, not a brand preference. Run your own prompts on 3-5 models and measure pass rates and cost rather than relying on reputation.