How to Run Qwen 3.8 27B Locally: VRAM, Quants and the Template Trap

The weights are out. Qwen 3.8 27B landed on Hugging Face on August 13, 2026, and the LICENSE file naming Apache 2.0 followed the next morning. The big Qwen 3.8 Max opened up a day earlier, but under Alibaba's own qwen3.8-max licence rather than Apache, and at 2.4 trillion parameters it is a data center model anyway. The 27B is the one people were actually waiting for: the one that fits on a card you can buy.

This guide is the measured version. Every file size below was read off the published repositories, not estimated from parameter counts, and the one bug that will waste your evening is in here too.

What Actually Landed

The architecture is inherited from Qwen 3.5, which matters more than it sounds: only 16 of the 64 layers use full attention, the other 48 are linear attention. That is what makes a 256K context realistic on a desktop, and we do the math on it below.

Will It Fit? Real File Sizes

Every number below is the file size Hugging Face reports for unsloth/Qwen3.8-27B-GGUF, read on August 14, 2026. Sizes differ by a few hundred megabytes between packs, so check the repo you actually download from rather than trusting any table, including this one.

Weights are only half the bill. The other half is the KV cache, and here the hybrid layout pays off hard.

The Context Math Nobody Prints

Only the 16 full attention layers keep a KV cache. Each has 4 KV heads at head dimension 256, so at fp16 one token costs 2 x 4 x 256 x 2 bytes = 4 KB per layer, 64 KB across all 16 layers.

For comparison, a conventional dense model with 64 full attention layers at the same head geometry would burn 256 KB per token, four times as much, and 32K context alone would cost 8 GB. That is the whole point of the Gated DeltaNet layers. A 24 GB card genuinely holds Q4_K_M with a 64K to 96K window, which no 27B could do a generation ago.

The Fastest Path: Model Manager

If you would rather not think about any of the above, Locally Uncensored reads your card, hides the quants that will not fit, and downloads the right one:

git clone https://github.com/PurpleDoubleD/locally-uncensored.git
cd locally-uncensored
# Windows: setup.bat | Linux: ./setup.sh

Or take the installer from the releases page, open the Model Manager and search for Qwen 3.8. Nothing leaves your machine and there is no account.

By Hand With llama.cpp

Use a current build. The trunk registers as the qwen35 architecture, so a llama.cpp from before the Qwen 3.5 family will refuse the file with an unknown architecture error rather than run it badly.

# text only
llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
  --ctx-size 32768 --n-gpu-layers 99 --jinja

# with vision
llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj Qwen3.8-27B-mmproj-bf16.gguf \
  --ctx-size 32768 --n-gpu-layers 99 --jinja

Two flags earn their place. --jinja makes llama.cpp use the model's own chat template instead of a generic one, which this model needs for its thinking blocks. --mmproj is the vision encoder; without it the model loads fine, answers text fine, and quietly ignores every image you paste, which is a confusing way to conclude that vision is broken.

The Template Trap

This one is worth the price of the article. The official jinja chat template wraps every assistant turn in a think block even when the reasoning is empty, then opens another think block when generation starts. In a single question and answer you never notice. In a multi turn conversation, and especially in an agent loop, the blocks nest and the history gets truncated. The model looks like it developed amnesia, and the natural conclusion is that the quantization is bad.

It is not the quantization. Several GGUF packs already ship a corrected chat_template.jinja and say so in their model cards. If you converted the weights yourself, or you pulled a GGUF in the first hours after release, replace the template before you debug anything else. Re download rather than patch by hand if the pack offers a fixed one.

One more day one detail: the official BF16 release contains a one layer MTP head for speculative decoding. Some quantizers strip it, because it adds roughly 0.5 to 0.8 GB and does nothing for answer quality. If you are not running --spec-type draft-mtp, a pack without the MTP head is strictly better for you.

Thinking Control

Thinking is on by default and the model exposes three effort levels, xhigh (the default), medium and low. Through an OpenAI compatible server:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="none")

r = client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[{"role": "user", "content": "Plan a migration from Postgres 14 to 17."}],
    reasoning_effort="medium",
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(r.choices[0].message.content)

Set enable_thinking to false for a direct answer. Sampling settings from the model authors: thinking mode wants temperature 1.0 and top_p 0.95, non thinking mode wants temperature 0.7 and top_p 0.80 with a presence penalty of 1.5. Those defaults matter more than usual on this family.

A warning from the model card that is easy to ignore: in multi turn agent work, lower effort is not automatically faster. Shallow analysis causes failures and retries, and the retries cost more time than the thinking would have.

On a Mac

MLX builds appeared the same day in 3, 5 and 6 bit, plus MXFP4 and NVFP4. A 6 bit MLX build sits around the Q6_K size, so 32 GB of unified memory is comfortable and 24 GB works at 4 bit. Apple Silicon shares memory between CPU and GPU, so the KV table above is the number to plan against, not a separate VRAM budget.

Uncensored Builds

Abliterated and heretic variants of Qwen 3.8 27B were published within hours of the weights. That is the usual pattern and it is the reason open weights matter: the hosted Qwen 3.8 Max carries the standard alignment and refusals, and a local copy carries whatever the community removed from it. If uncensored is the point, see the abliterated models guide for what those builds change and what they cost in accuracy, and the 2026 roundup for the alternatives.

Qwen 3.8 27B or Qwen 3.6 27B?

If you already run Qwen 3.6 and it does what you need, there is no emergency. The 3.8 is the better model at the same size and the same license, with a longer native context and stronger agent behaviour, but it is a 17.1 GB download and a template gotcha. Upgrade when you have an evening, not in the middle of a project.

If you are on 8 GB or 12 GB, the honest answer is that this is the wrong size for your card. A 2-bit quant technically loads at 10.7 GB and leaves you no context. A smaller Qwen will make you happier.

FAQ

How much VRAM does Qwen 3.8 27B need?

17.1 GB for the Q4_K_M weights, plus 64 KB per token of context. A 24 GB card is the comfortable home. 32 GB gets you Q8_0.

Does it fit on an RTX 3060 12 GB?

Only at 2-bit, about 10.7 GB, with almost nothing left for context. It runs, slowly, with layers offloaded to system RAM.

Is it really Apache 2.0?

Yes. Commercial use, fine tuning and redistribution are all permitted.

Why does my chat get cut off after a few turns?

The official chat template nests think blocks and truncates history. Use a GGUF pack with the corrected template.

Can it see images?

Yes, with the separate 0.9 GB mmproj file loaded alongside the weights. Without it, images are silently ignored.

How do I turn thinking off?

Pass enable_thinking: false in chat_template_kwargs, or lower reasoning_effort from xhigh to medium or low.

Getting Started

The whole point of an Apache 2.0 release is that you do not have to ask anyone. Pull the app, pick a quant that fits your card, and the model runs on your own hardware with no account, no logging and no refusals you did not choose:

git clone https://github.com/PurpleDoubleD/locally-uncensored.git
cd locally-uncensored
# Windows: setup.bat | Linux: ./setup.sh

Locally Uncensored is AGPL-3.0 licensed and free to use. Built by PurpleDoubleD.