Run an LLM and Stable Diffusion Together on One GPU

Yes — one GPU can run both a chat LLM and Stable Diffusion. What it usually cannot do is hold both in VRAM at the same time. The fix is sequential loading: the model you are using right now occupies the GPU, and the other one is unloaded until you switch. This guide covers the honest VRAM math, the three ways to set that up, and which model pairs actually work on 8, 12, 16 and 24 GB cards.

The Real Problem: VRAM Math

Chat models and image models are both VRAM-hungry, and their requirements add up. Rough numbers — deliberately ballpark, because quantization, resolution and context length move them, but not enough to change the conclusion:

Now add them. A 7B-class chat model at Q4 needs 4–5 GB; SDXL-class image generation wants 7–10 GB while rendering. Combined, that is 12–15 GB — already past a 12 GB card, and we have not counted the KV cache that grows with every conversation turn or the few hundred megabytes Windows reserves for the desktop itself. Step up to a 14B chat model (~9 GB) and there is no image model it coexists with on consumer hardware.

So the question is not really “can my GPU run both?” — it is “who unloads the model I am not using right now?” There are three answers.

Three Ways to Run Both

Option 1: Keep Both Loaded (24 GB+)

If you have a 24 GB card, the problem mostly disappears: a 5 GB chat model and an 8 GB image model sit in VRAM together with room left for context and rendering. Zero switching cost, simplest mental model. Below 24 GB this stops being fun — a small pair technically squeezes into 16 GB, but the first long conversation or high-resolution render pushes you into out-of-memory territory. The other genuine both-at-once setup is two GPUs, one per engine, which works but is its own project.

Option 2: Two Tools, Manual Stop/Start

The classic setup: Ollama serves chat from the terminal, ComfyUI or Automatic1111 generates images in a browser tab, both pointed at the same GPU. This genuinely works — Ollama unloads an idle model after about five minutes by default, and ComfyUI loads and unloads models as needed. If you already know both tools, it is a legitimate way to run them, and our ComfyUI beginners guide covers the image side in depth.

The cost is that you are the memory manager. Two installations, two interfaces, and the gap between them: generate an image while the chat model is still warm and you get a crawling render or a CUDA out-of-memory error. So you wait out the keep-alive, run ollama stop, or tune keep_alive yourself — every time you switch. Functional, but it turns “draw me this” into a small operations task.

Option 3: One Studio That Handles the Juggling

Locally Uncensored is an all-in-one local AI studio — chat, image, video and a coding agent in one installer — and it was built around exactly this constraint. Both engines live in one app on the same GPU, so switching between Chat and Create does not mean stopping one tool and starting another. The app installs and manages ComfyUI itself (or connects to your existing install), and the image backend loads and unloads its models as needed. On the chat side, every model has a power icon in the model selector: green means loaded, one click unloads it from VRAM — no terminal, no restarting anything.

It also keeps you from pairing models that can never fit: the Discover tab filters its one-click downloads by VRAM tier (Lightweight ≤10 GB, Mid-Range 10–16 GB, High-End >16 GB) and recommends models based on your actual hardware.

Step by Step: Chat and Images in One App (Windows)

Step 1: Install Locally Uncensored

Download the installer from GitHub Releases — the current v2.5.10 installer is about 7 MB — and install it like any normal Windows app. No Docker, no command line, no config files. Linux builds (deb, rpm, AppImage) are on the same releases page.

Step 2: Let the Wizard Handle the Engines

On first launch the setup wizard scans for 12 local backends (Ollama, LM Studio, llama.cpp and others) and for existing ComfyUI installs, including the ComfyUI Desktop App layout. Whatever you already have keeps working; whatever is missing, the app sets up with one click.

Step 3: Pick a Chat Model That Fits

Open the Model Manager and filter by your VRAM tier. On an 8 GB card, a 7–8B GGUF at Q4 (~5 GB) is the right size; on 12 GB, a 14B (~9 GB) is the sweet spot. If you want models that answer without lectures, the abliterated models guide has the best picks per VRAM class — all one-click downloads.

Step 4: Pick an Image Model in Create

Switch to the Create tab and install an image bundle: Juggernaut XL (one ~7 GB file, SDXL class, photorealism) runs on GPUs with 6+ GB VRAM; FLUX 2 Klein wants 8–10 GB. The app builds the ComfyUI workflows automatically — you get a prompt box and a Generate button, not a node graph.

Step 5: Switch Freely

Chat in the Chat tab, generate in Create, or just ask the chat for a picture and it draws one in the conversation — the image model is loaded for the render. The image backend frees what it no longer needs, and if you want the chat model out of VRAM immediately, its power icon unloads it in one click. What you never do: watch nvidia-smi, restart ComfyUI, or juggle a terminal window next to a browser tab.

What Pairs Work: 8 / 12 / 16 / 24 GB

All pairings below assume sequential use — one model in VRAM at a time, which is how a single person actually works. On 24 GB, the smaller pairs can stay resident together.

One more honest number from the video side: budget 32 GB of system RAM if video generation is part of your plan — the models need working memory beyond what sits on the GPU. For chat plus images, 16 GB of RAM is fine.

FAQ

Can I run an LLM and Stable Diffusion at the same time?

Truly at the same time — both resident in VRAM — needs their combined footprint, which in practice means a 24 GB card for a useful pair, or two GPUs. On ordinary 8–16 GB cards you run them sequentially on the same GPU: the model you are using sits in VRAM, the other is unloaded, and switching takes seconds. For one person alternating between chatting and generating, that feels close to simultaneous.

How much VRAM do I need for both?

8 GB is the honest entry point: a 7–8B chat model plus SDXL-class images, sequentially. 12 GB is the sweet spot — a 14B chat model plus FLUX 2 Klein or Z-Image Turbo, and entry-level video. 24 GB removes the juggling entirely for small pairs.

Does Ollama work with ComfyUI on one GPU?

Yes — that is Option 2 above. Ollama idle-unloads after about five minutes by default and ComfyUI loads models as needed, so they can share a GPU; the friction is coordinating the handover yourself whenever you alternate quickly. Locally Uncensored auto-detects existing Ollama and ComfyUI installs and fronts both from one interface, so your downloaded models keep working. The full engine-vs-studio comparison is on the Ollama page.

What about video models?

Video is the heaviest lane: plan on 10–12 GB VRAM for text-to-video, while FramePack F1 does image-to-video on 6 GB. The same sequential logic applies — the video model gets the GPU while it renders. Our local video guide covers Wan 2.2 on a 12 GB card step by step.

Do I have to reinstall my models if I already use Ollama or ComfyUI?

No. The app auto-detects 12 local backends and scans common paths for ComfyUI, including the Desktop App layout. Existing models and checkpoints keep working; you are only changing which interface sits in front of them.

Next Steps

• ComfyUI beginners guide — what the image engine does under the hood
• Abliterated models guide — the best uncensored chat model for your VRAM
• Generate AI videos locally — Wan 2.2 and HunyuanVideo on a 12 GB GPU