Production AI on an 8-12 GB card
An ordinary 8-12 GB GPU — the RTX 3060 12G, 4060, 2060 Super and their equivalents already in millions of machines — now runs models good enough for real, bounded work. This is the deployment playbook: which models fit, how to stand one up behind an API, and what you gain and lose against a subscription.
What “production-grade” means at this tier
Not frontier coding. Reliability on a bounded task set.
A 12 GB card will not replace Opus or Astra on long-horizon engineering — the v5 results put even the best released open-weight models 26 points behind the leader. What it does well, at zero marginal cost and full privacy: classification and routing, extraction from documents, drafting and rewriting, summarisation, small single-file code fixes, agent glue between expensive steps. Production-grade here means the same task finishes the same way every time, inside a context window you control, with no rate limit and no invoice. The capability ceiling is real; pick tasks under it.
The October 2026 shortlist
Coding scores are DeepSWE pass-at-one figures with the measurement source labelled; general scores are benchlm community figures. VRAM is the quantised footprint plus room for the KV cache.
| Model | VRAM at shipped quant | Fits | Best at | DeepSWE coding | General score |
|---|---|---|---|---|---|
| Gemma 4 12B QAT (Google) | ~7.2 GB at Q4_0 (quantisation-aware trained, not post-hoc) | 8-12 GB | general text, multimodal input, drafting | no run | 52 (benchlm) |
| Zyphra ZAYA1-8B | ~5 GB at 4-bit | 8 GB | the strongest pure-text 8 GB class right now | no run | 55 (benchlm) |
| Mellum2-12B-A2.5B-Thinking | ~7 GB (MoE, 2.5B active) | 8-12 GB | reasoning-heavy work at low latency | no run | thinking specialist (community) |
| gpt-oss-20b (OpenAI) | ~12-14 GB; MoE with 3.6B active | 16 GB tier | the step up if you have a 16 GB card | no run | strong tool-use record |
| Qwen3.8-27B (Alibaba) | 14-17 GB at 4-bit | 24 GB card | the best single-card coder you can run; out of reach for 12 GB | 42.2 (independent) | — |
| Qwen3.8-Flash-Next (Alibaba) | ~12 GB at IQ3_S through the Strata engine: 125B MoE with 6B active per token | 12 GB | agentic coding; the base model behind Qwen3.8 Flash | 58.7 (vendor figure, no independent listing) | no v5 figure listed here |
| DeepSeek V4.1 Flash — reference row, does not fit | 284B-class MoE; server class | no | bounded code edits, API tier | 74.2 vendor / 65.6 under Codex | 32 on AA v5, below Claude Haiku 5.5 |
Trend note: late-2026 releases increasingly ship 2-bit and QAT checkpoints as first-class options, which is what keeps 12 GB-class models useful as they grow. Source labels: benchlm.ai community tracker; runaihome.com on the 2-bit shift. The DeepSWE figure for Qwen3.8-27B is the independent table, not a vendor claim.
Deploy it behind an API in six steps
Works the same on Windows, macOS and Linux.
- Install a runtime. Ollama (CLI + service) or LM Studio (GUI + server). Both expose an OpenAI-compatible endpoint: Ollama on
localhost:11434, LM Studio onlocalhost:1234. - Do the VRAM math before choosing. Model file size + 1-2 GB for the KV cache at 8K context. A 7.2 GB model on a 12 GB card is right; on an 8 GB card choose the 8B class instead.
- Pull the quantised build. Prefer a QAT or 2-bit release over re-quantising an FP16 download: same fit, better quality.
- Pin the serving settings. Fixed system prompt, temperature 0.2-0.4 for production work, context length set to what you actually send, keep-alive on so the model stays resident.
- Point your tools at it. Anything that speaks the OpenAI API works: agent harnesses, scripts, IDE extensions. Set the base URL to localhost and a dummy key.
- Earn the word production. Before trusting it: run 20 of your own real tasks through it, add a timeout and one retry, and validate the output shape (JSON parse, file exists, tests pass) instead of trusting the model’s own report.
A coding score is not office capability: two strata stories
The same model can sit near the top of a coding board and below a small closed model on general intelligence. Two October 2026 cases show exactly where that boundary runs — and why the shortlist above now lists the two scores separately.
| Model | The coding claim (public test) | The private-test reality | Reading |
|---|---|---|---|
| DeepSeek V4.1 Flash (the 1017-era refresh) | 74.2 on the maker’s model card; cross-harness runs span 65.5 to 74.2, 65.6 under Codex; the first open-weight model past GPT-5.3 Codex’s 70 on the home card | Artificial Analysis v5 total: 32, below Claude Haiku 5.5’s 36. FrontierCode 2.0 (132 private tasks, graded by maintainer merges): 28%. Terminal-Bench-Science: 13%. Terminal-Bench 3.0: 30.0 against Opus 5’s 43.3 | The refresh optimised agentic coding behaviour on a task set anyone can download; the held-answer sets show the rest of the profile did not move |
| Qwen3.8-Flash-Next | 58.7 published by Alibaba; no independent listing yet | no v5 figure listed here | A claim, not a measurement, until a second harness runs it. The Strata notes below are about fit, not proof of capability |
Why the 1017 coding number rose while its v5 total stayed at 32
Three mechanisms, in order of how much they explain.
- The refresh is specialised post-training. V4 Flash sat at 53.3 on this board; the 1017-era update concentrates on agentic coding loops — edit, run tests, iterate — which is exactly what DeepSWE measures. A narrow RL pass can move that one number a lot.
- DeepSWE is public. Its tasks and harness are reproducible, so a lab can post-train against them. The vendor figure is taken on the maker’s own harness; the same weights span 65.5 to 74.2 across harnesses, which means harness choice moves the score almost as much as model quality does. See the contamination analysis.
- v5 moves 60% of scoring to private held-answer tests. Knowledge, science and general categories pull the total down to 32. On the private coding test the model still leads open weights at 28% — so the capability is real but narrow: 18 points behind Opus on FrontierCode, 13% on research-science terminals, below Haiku on the total.
Deployment translation: route bounded code edits, refactors and test-fixing to the 1017 tier, because that is the stratum its training actually moved. Do not route document work, knowledge questions, analysis or science tasks to it — those need a model strong on the general categories, and the v5 total of 32 says this one is not.
Strata field notes: Qwen3.8-Flash-Next on a 12 GB card
Deployment notes from an RTX 5070 12 GB build running the Strata engine — what fits, and where the boundary is.
- Why it fits at all: 125B total parameters but 6B active per token, so an IQ3_S build lands inside 12 GB. This is the cheapest route today to a 58.7-class coding claim on an ordinary card.
- Context ceiling: 256K native locally. The advertised 1M context exists only on the cloud-hosted build; a local deployment inherits neither it nor the server’s throughput.
- Engine lock-in: Strata runs the qwen35moe family only (Qwen3.8-Flash-Next, Qwen3.6-35B-A3B). DeepSeek, Llama, gpt-oss and Qwen dense variants cannot run on it — so the 1017 model above is not even an option in this stack.
- Chinese production work: use the full IQ3_S build, not the Coder variant; the Coder build mis-generates and repeats on Chinese output.
- Vision: on a 12 GB card, CPU vision runs roughly 3x faster than on-GPU; pass
--visionexplicitly because the default install disables it. - Housekeeping: run
--calibrateafter deployment for the hardware, use k8v4 KV streaming for long context, and on Alder Lake-or-later CPUs without AVX-512 setSTRATA_IQ3S_MT1=1.
Pros and cons, stated plainly
The trade is capability for control.
| Dimension | 8-12 GB local | Frontier API / subscription |
|---|---|---|
| Cost | one-time hardware; ~zero per token | $20/mo+ or per-token; Opus 5.5 runs $10.04 per completed task on AA’s measurement |
| Capability ceiling | benchlm 52-55 class; bounded tasks only | 58-79 tier; long-horizon engineering |
| Privacy | nothing leaves the machine | depends on the vendor’s training terms |
| Rate limits | none; yours at 3 AM | caps, peak pricing, outages |
| Context | 8-32K realistically on 12 GB | 200K-1M |
| Maintenance | you own updates, quant choice, failures | vendor’s problem |
| Tools | no server-side search or code execution | built-in harness features |
Back to local Coding tier guide Hardware and video comparison