~ / local / 8-12 GB playbook

Production AI on an 8-12 GB card

An ordinary 8-12 GB GPU — the RTX 3060 12G, 4060, 2060 Super and their equivalents already in millions of machines — now runs models good enough for real, bounded work. This is the deployment playbook: which models fit, how to stand one up behind an API, and what you gain and lose against a subscription.

What “production-grade” means at this tier

Not frontier coding. Reliability on a bounded task set.

A 12 GB card will not replace Opus or Astra on long-horizon engineering — the v5 results put even the best released open-weight models 26 points behind the leader. What it does well, at zero marginal cost and full privacy: classification and routing, extraction from documents, drafting and rewriting, summarisation, small single-file code fixes, agent glue between expensive steps. Production-grade here means the same task finishes the same way every time, inside a context window you control, with no rate limit and no invoice. The capability ceiling is real; pick tasks under it.

The October 2026 shortlist

Coding scores are DeepSWE pass-at-one figures with the measurement source labelled; general scores are benchlm community figures. VRAM is the quantised footprint plus room for the KV cache.

ModelVRAM at shipped quantFitsBest atDeepSWE codingGeneral score
Gemma 4 12B QAT (Google)~7.2 GB at Q4_0 (quantisation-aware trained, not post-hoc)8-12 GBgeneral text, multimodal input, draftingno run52 (benchlm)
Zyphra ZAYA1-8B~5 GB at 4-bit8 GBthe strongest pure-text 8 GB class right nowno run55 (benchlm)
Mellum2-12B-A2.5B-Thinking~7 GB (MoE, 2.5B active)8-12 GBreasoning-heavy work at low latencyno runthinking specialist (community)
gpt-oss-20b (OpenAI)~12-14 GB; MoE with 3.6B active16 GB tierthe step up if you have a 16 GB cardno runstrong tool-use record
Qwen3.8-27B (Alibaba)14-17 GB at 4-bit24 GB cardthe best single-card coder you can run; out of reach for 12 GB42.2 (independent)—
Qwen3.8-Flash-Next (Alibaba)~12 GB at IQ3_S through the Strata engine: 125B MoE with 6B active per token12 GBagentic coding; the base model behind Qwen3.8 Flash58.7 (vendor figure, no independent listing)no v5 figure listed here
DeepSeek V4.1 Flash — reference row, does not fit284B-class MoE; server classnobounded code edits, API tier74.2 vendor / 65.6 under Codex32 on AA v5, below Claude Haiku 5.5

Trend note: late-2026 releases increasingly ship 2-bit and QAT checkpoints as first-class options, which is what keeps 12 GB-class models useful as they grow. Source labels: benchlm.ai community tracker; runaihome.com on the 2-bit shift. The DeepSWE figure for Qwen3.8-27B is the independent table, not a vendor claim.

Deploy it behind an API in six steps

Works the same on Windows, macOS and Linux.

  1. Install a runtime. Ollama (CLI + service) or LM Studio (GUI + server). Both expose an OpenAI-compatible endpoint: Ollama on localhost:11434, LM Studio on localhost:1234.
  2. Do the VRAM math before choosing. Model file size + 1-2 GB for the KV cache at 8K context. A 7.2 GB model on a 12 GB card is right; on an 8 GB card choose the 8B class instead.
  3. Pull the quantised build. Prefer a QAT or 2-bit release over re-quantising an FP16 download: same fit, better quality.
  4. Pin the serving settings. Fixed system prompt, temperature 0.2-0.4 for production work, context length set to what you actually send, keep-alive on so the model stays resident.
  5. Point your tools at it. Anything that speaks the OpenAI API works: agent harnesses, scripts, IDE extensions. Set the base URL to localhost and a dummy key.
  6. Earn the word production. Before trusting it: run 20 of your own real tasks through it, add a timeout and one retry, and validate the output shape (JSON parse, file exists, tests pass) instead of trusting the model’s own report.

A coding score is not office capability: two strata stories

The same model can sit near the top of a coding board and below a small closed model on general intelligence. Two October 2026 cases show exactly where that boundary runs — and why the shortlist above now lists the two scores separately.

ModelThe coding claim (public test)The private-test realityReading
DeepSeek V4.1 Flash (the 1017-era refresh)74.2 on the maker’s model card; cross-harness runs span 65.5 to 74.2, 65.6 under Codex; the first open-weight model past GPT-5.3 Codex’s 70 on the home cardArtificial Analysis v5 total: 32, below Claude Haiku 5.5’s 36. FrontierCode 2.0 (132 private tasks, graded by maintainer merges): 28%. Terminal-Bench-Science: 13%. Terminal-Bench 3.0: 30.0 against Opus 5’s 43.3The refresh optimised agentic coding behaviour on a task set anyone can download; the held-answer sets show the rest of the profile did not move
Qwen3.8-Flash-Next58.7 published by Alibaba; no independent listing yetno v5 figure listed hereA claim, not a measurement, until a second harness runs it. The Strata notes below are about fit, not proof of capability

Why the 1017 coding number rose while its v5 total stayed at 32

Three mechanisms, in order of how much they explain.

  1. The refresh is specialised post-training. V4 Flash sat at 53.3 on this board; the 1017-era update concentrates on agentic coding loops — edit, run tests, iterate — which is exactly what DeepSWE measures. A narrow RL pass can move that one number a lot.
  2. DeepSWE is public. Its tasks and harness are reproducible, so a lab can post-train against them. The vendor figure is taken on the maker’s own harness; the same weights span 65.5 to 74.2 across harnesses, which means harness choice moves the score almost as much as model quality does. See the contamination analysis.
  3. v5 moves 60% of scoring to private held-answer tests. Knowledge, science and general categories pull the total down to 32. On the private coding test the model still leads open weights at 28% — so the capability is real but narrow: 18 points behind Opus on FrontierCode, 13% on research-science terminals, below Haiku on the total.

Deployment translation: route bounded code edits, refactors and test-fixing to the 1017 tier, because that is the stratum its training actually moved. Do not route document work, knowledge questions, analysis or science tasks to it — those need a model strong on the general categories, and the v5 total of 32 says this one is not.

Strata field notes: Qwen3.8-Flash-Next on a 12 GB card

Deployment notes from an RTX 5070 12 GB build running the Strata engine — what fits, and where the boundary is.

Pros and cons, stated plainly

The trade is capability for control.

Dimension8-12 GB localFrontier API / subscription
Costone-time hardware; ~zero per token$20/mo+ or per-token; Opus 5.5 runs $10.04 per completed task on AA’s measurement
Capability ceilingbenchlm 52-55 class; bounded tasks only58-79 tier; long-horizon engineering
Privacynothing leaves the machinedepends on the vendor’s training terms
Rate limitsnone; yours at 3 AMcaps, peak pricing, outages
Context8-32K realistically on 12 GB200K-1M
Maintenanceyou own updates, quant choice, failuresvendor’s problem
Toolsno server-side search or code executionbuilt-in harness features
The honest summary. Token freedom on this tier is real: classification, extraction, drafting and glue work cost you nothing per call and never leave your machine. What it is not: a substitute for a frontier model on long-horizon coding. The cheapest production pattern in late 2026 is local-first for the boring 80% and a pay-per-task frontier seat for the hard 20%.

Back to local Coding tier guide Hardware and video comparison