Running DeepSeek on an RTX 3090: How MoE Beats the 24 GB Limit
Running DeepSeek on an RTX 3090: what sits in VRAM, what goes to RAM, how big the files are and what sets the speed. Explained with open V3 and R1 numbers.
Large DeepSeek models run on a single RTX 3090 thanks to their mixture of experts (MoE) architecture. Only a small share of the weights works on each token, so the GPU holds the fixed part of the model and the context cache, while most of the experts sit in ordinary system RAM and are computed by the CPU. Whether a model fits comes down to a simple sum: file size against 24 GB of VRAM plus free RAM. How fast it answers depends on system memory bandwidth. DeepSeek V3 and R1 show all of this clearly: their weights, configuration and technical report are public, so every number can be recomputed.
KRAKOBIT covered the specific case of a new model on an RTX 3090 with 64 GB of RAM in a news piece on DeepSeek V4.1 Flash. This article covers the mechanics behind such setups.
What mixture of experts means at DeepSeek
In a regular dense model every token passes through every weight. A 70 billion parameter model reads all 70 billion for one piece of a word. In an MoE model some layers are split into many small subnetworks called experts. A router picks a few of them for each token, and the rest stay idle.
DeepSeek has followed this path since January 2024. The lineup:
| Model | Date | Total parameters | Active per token | Active share |
|---|---|---|---|---|
| DeepSeekMoE 16B | January 2024 | 16.4B | 2.8B | 17% |
| DeepSeek V2 | May 2024 | 236B | 21B | 9% |
| DeepSeek V2-Lite | May 2024 | 15.7B | 2.4B | 15% |
| DeepSeek V3 | December 2024 | 671B | 37B | 5.5% |
| DeepSeek R1 | January 2025 | 671B | 37B | 5.5% |
The last column shows the trend: the models grow while the share of working weights shrinks. For home use this is the key property. A huge model reads only a small piece of itself per step, so most of it can live in memory that is roomy but slower.
How one DeepSeek V3 layer is built
According to the DeepSeek V3 technical report and the model configuration file on Hugging Face, V3 has 61 layers. The first three are ordinary dense layers. In the other 58, a single large block is replaced by 257 experts: one shared expert that every token passes through and 256 routed experts, of which 8 are chosen per token.
The size of one expert follows from the configuration: hidden size 7168, expert intermediate size 2048, three matrices. That gives about 44 million parameters per expert (estimate based on config.json).
The arithmetic from there:
- routed experts working on one token: 8 × 58 layers × 44M ≈ 20B parameters (estimate);
- everything else that runs on every token (attention, shared experts, the three dense layers, embeddings): about 17B (estimate, the gap between 37B and the routed part);
- routed experts waiting their turn: about 650B out of 671B.
That defines the split. Around 17B parameters are needed at every step, so it pays to keep them on the GPU. Around 650B are needed rarely and in turns, so they can live in RAM. Each token reads about 20B of them.
What goes into VRAM and what goes into RAM
The GPU gets:
- attention blocks, which link the current token to the text before it;
- the router;
- the shared expert of each layer;
- the three dense layers at the start of the model;
- the context cache, the model’s memory of the conversation so far.
System RAM gets the routed experts. The CPU computes the selected experts right where they sit. Only small intermediate vectors travel over the PCIe bus between GPU and CPU, not gigabytes of weights.
In llama.cpp, the open engine for running models on ordinary computers, this layout is set with -ot exps=CPU: every tensor with exps in its name stays on the CPU. Later the engine gained the shorter --cpu-moe and --n-cpu-moe N options. The second keeps experts from only N layers on the CPU and hands the rest to the GPU if it has room. The same principle was demonstrated in early 2025 by the KTransformers project from Tsinghua University: full-size V3 and R1 on a single 24 GB GPU with hundreds of gigabytes of RAM in a server.
Why MLA cache compression makes 24 GB enough
VRAM has two consumers: weights and the context cache. The cache grows with every token of the conversation. In ordinary models it is what eats video memory on long documents.
With DeepSeek V2 the company introduced Multi-head Latent Attention (MLA). Instead of full keys and values for each attention head, the model stores one compressed vector per token in each layer. According to the V2 paper, this cut the cache by 93.3% compared with the earlier DeepSeek 67B.
For V3 the numbers work out like this. Each layer stores 576 values per token: 512 for the compressed vector and 64 for the positional part. Across 61 layers that is 35,136 values, about 70 KB per token at 16 bits each. A 32,000-token context takes about 2.2 GB, a 128,000-token context about 9 GB (estimate from the configuration, excluding app buffers). Without MLA, with 128 attention heads, the same model would store dozens of times more.
llama.cpp gained support for the compressed MLA cache in spring 2025. Before that the engine expanded the cache to full size, and long DeepSeek contexts ate VRAM far faster.
How big the files are and what fits in 24 GB plus 64 GB
A model file’s size equals the parameter count times bits per parameter, divided by eight. The original V3 weights are stored in 8-bit FP8 and, together with an extra token prediction module, take about 690 GB.
Home builds compress harder. The best-known compressed R1 builds came from the Unsloth team in January 2025. Their trick is dynamic quantization: attention and shared parts stay at 4 to 6 bits, while routed experts drop to 1.58 bits. By Unsloth’s own account, uniformly squeezing the whole model to 1.58 bits produced gibberish and endless repetition.
| DeepSeek R1 build | File size |
|---|---|
| Original FP8 | about 690 GB |
| Unsloth 2.51-bit | 212 GB |
| Unsloth 2.22-bit | 183 GB |
| Unsloth 1.73-bit | 158 GB |
| Unsloth 1.58-bit | 131 GB |
An RTX 3090 with 64 GB of RAM has 88 GB in total. The operating system, a browser and the app itself take 6 to 10 GB of RAM (estimate), leaving roughly 78 to 82 GB for the model. Full-size V3 and R1 do not fit even in the smallest build: they need at least 24 GB of VRAM and 128 GB of RAM, and 192 GB to be comfortable.
If the file is bigger than memory, llama.cpp can read the missing weights straight from the SSD. The model will start, but every token waits on the drive, and speed falls to fractions of a token or a few tokens per second depending on the disk.
That sets the practical ceiling for a 3090 with 64 GB: MoE models whose files fit in about 80 GB. At 4-bit that means models up to roughly 140B total parameters (estimate). At the other end sits V2-Lite: about 10 GB at 4 bits, it fits entirely in VRAM and never touches system RAM.
What sets response speed
Text generation is limited by memory reads, not compute. Every token requires reading all active weights. The part in the RTX 3090’s VRAM is read at 936 GB/s. The part in system RAM is read by the CPU 9 to 18 times slower: dual-channel DDR4-3200 gives a theoretical 51.2 GB/s, DDR5-6400 gives 102.4 GB/s. The slow part sets the pace.
The speed ceiling is RAM bandwidth divided by the amount of expert data read from it per token. At 4-bit, with overhead, one parameter takes about 0.56 bytes.
| Routed parameters per token from RAM | Read per token | Ceiling on DDR4-3200 | Ceiling on DDR5-6400 |
|---|---|---|---|
| 3B | 1.7 GB | 30 tok/s | 60 tok/s |
| 10B | 5.6 GB | 9 tok/s | 18 tok/s |
| 20B (like V3) | 11.2 GB | 4.6 tok/s | 9 tok/s |
Every figure in the table is an estimate of the theoretical ceiling. In practice a CPU pulls 50 to 70% of rated bandwidth from memory (estimate), so real numbers will be lower. The table shows the proportion: speed is set by the model’s active parameters and the RAM type, not by GPU power. Moving from DDR4 to DDR5 nearly doubles the ceiling in this setup. Swapping the RTX 3090 for a card with the same memory size barely moves it.
At 1.58 bits, V3’s per-token experts weigh about 4 GB instead of 11 (estimate), which is why servers with multi-channel memory reach double-digit speeds on it. Eight-channel DDR5 on a server platform delivers 300 to 400 GB/s, a different class of machine.
Why the model takes a while to read a long document
A local model has two speeds. The first, generation, is covered above. The second is reading the input: before answering, the model processes the entire loaded document.
Here large batches of tokens are processed at once, and llama.cpp finds it cheaper to move expert weights to the GPU over PCIe and compute there than to run them on the CPU. The RTX 3090 uses PCIe 4.0 x16, about 32 GB/s in each direction. The more experts and the longer the document, the longer the pause before the first word. On an older board with PCIe 3.0 the bus is half as fast.
Why the RTX 3090 specifically
The RTX 3090 came out in September 2020: 24 GB of GDDR6X, 936 GB/s, a rated 350 W. It is the oldest NVIDIA consumer card with 24 GB, which makes it the cheapest card of that size on the second-hand market. In an MoE setup memory size matters more than core speed: 24 GB holds the fixed part of the model and a long context cache, while the experts are read from RAM anyway.
A different route, two GPUs of different classes in one machine, was covered by KRAKOBIT using a local Qwen on an RTX 5090 plus 5060 Ti.
How a local DeepSeek differs from the cloud version
The weights are the same ones running at DeepSeek and at resellers like OpenRouter, where the company’s models recently took more than half of all tokens in a day, as KRAKOBIT reported in its OpenRouter story. The differences lie in three places.
First, precision. The cloud runs the original 8-bit weights, while home setups almost always use a compressed build. The lower the bit count, the more often the model slips in code, arithmetic and long reasoning.
Second, speed. Server accelerators hold the whole model in fast memory and serve hundreds of requests at once. A home card serves one user.
Third, the network. Once downloaded, the model file works without a single call to DeepSeek’s servers: no account, no key, no payment, no country check. When Iran shut down the internet nationwide in June 2025, cloud AI services were unreachable, while a model on disk would have kept working. Internet access is needed only to download the app and the model file.
What works on an RTX 3090 and what does not
Works:
- small DeepSeek MoE models like V2-Lite entirely on the GPU, fast;
- MoE models with files up to 80 GB on a 3090 plus 64 GB, with experts in RAM;
- long context on MLA models with a recent llama.cpp;
- offline use after download.
Does not work or works poorly:
- full-size V3 and R1 on 64 GB of RAM: the file is bigger than memory, leaving only SSD reads;
- dense models larger than 24 GB: they need every weight on every token, and RAM slows each step;
- uniform quantization below 2 bits without dynamic builds: quality collapses into incoherent text;
- several people sharing one card at the same time.
For any new DeepSeek MoE model, the check comes down to three numbers from its model card: total parameters give the file size, active parameters give the speed ceiling, and MLA tells how much context fits in 24 GB.