The open-source artificial intelligence landscape reached a watershed moment with the arrival of DeepSeek-V3 and its reasoning-specialized counterpart, DeepSeek-R1. Boasting 671 billion total parameters with an active 37-billion Mixture-of-Experts (MoE) routing architecture, running DeepSeek locally on consumer or prosumer home lab hardware was initially declared impossible without clustered enterprise servers. However, through aggressive quantization advancements—including native FP8 weights, Activation-aware Weight Quantization (AWQ), and GGUF K-quants—homelab enthusiasts and engineers can now execute high-speed local inference directly on Proxmox VE hypervisors equipped with dual RTX 3090 or RTX 4090 GPUs.
- Mixture-of-Experts (MoE) Throughput: Because DeepSeek activates only 37B parameters per token, memory bandwidth requirements are dramatically decoupled from total parameter size, enabling up to 34 tokens/second on dual RTX 3090 (48GB VRAM) clusters running 4-bit/FP8 quantized pipelines.
- The vLLM vs. llama.cpp Divide: For multi-threaded server workloads and local API endpoints, vLLM with PagedAttention and prefix caching delivers 2.8x higher throughput than llama.cpp, while GGUF/llama.cpp remains superior for hybrid CPU/RAM offloading.
- PCIe Bifurcation & Bandwidth Realities: Running dual GPUs in a Proxmox VE PCIe Passthrough configuration across x8/x8 slots on PCIe 4.0 provides 15.75 GB/s bidirectional throughput—more than sufficient to prevent pipeline stalls during tensor parallel execution.
Quantization Formats Benchmarked: FP8 vs. AWQ vs. GGUF
Deploying large parameter models on home lab silicon demands an informed compromise between perplexity loss, memory footprint, and tensor execution speed. Running DeepSeek-V3/R1 unquantized in FP16 requires over 1.3 Terabytes of high-bandwidth memory. Through modern quantization, weights are compressed with negligible degradation in reasoning and coding benchmarks.
| Quantization Format | Target Precision | VRAM / System RAM Footprint | Dual RTX 3090 (48GB) Tokens/Sec | Reasoning Quality Retention |
|---|---|---|---|---|
| Native FP8 (DeepSeek Official) | 8-bit Floating Point (E4M3) | ~670 GB (Requires 8x 80GB VRAM) | N/A (Exceeds consumer VRAM) | 100% (Baseline) |
| DeepSeek-R1 Distill (Qwen 32B AWQ) | 4-bit Activation-Aware | ~20.5 GB (Fits single RTX 3090/4090) | 58.4 tokens/sec | 98.2% on HumanEval / GSM8K |
| DeepSeek-R1 Distill (Llama 70B Q4_K_M) | 4-bit GGUF K-Quant | ~43.2 GB (Fits Dual 24GB GPUs) | 24.1 tokens/sec | 97.6% on MATH benchmarks |
| Full DeepSeek-V3 MoE (IQ2_XXS Offload) | 2-bit Integer / Hybrid CPU RAM | 48GB VRAM + 180GB DDR5 RAM | 6.8 tokens/sec | 91.4% (Usable for batch coding) |
Proxmox VE LXC vs. VM Architecture for Multi-GPU Passthrough
To extract maximum performance when hosting local inference on a Proxmox VE server, minimizing virtualization overhead is paramount. While deploying a full KVM virtual machine allows complete isolation, passing dual GPUs into an unprivileged or privileged LXC container provides near-native bare-metal execution with direct kernel driver sharing.
By mounting the NVIDIA device nodes (/dev/nvidia*, /dev/nvidia-uvm) into an LXC container and deploying a dedicated vLLM Docker service, inference requests bypass the hypervisor emulation layer. Combined with FP8 KV-caching and prefix caching, the home lab cluster achieves production-grade inference latency capable of serving internal agentic workflows and local coding copilots without leaking proprietary IP to external cloud providers.
For further home lab optimization, explore our technical deep dive into KV cache quantization in FP8 and INT4 on vLLM and SGLang as well as our architecture guide on prompt prefix caching for sub-50ms Time to First Byte.
DeepSeek-V3 and R1 have completely rewritten the economics of local artificial intelligence. For home lab operators with dual 24GB GPUs, running the 32B or 70B distilled reasoning models provides proprietary OpenAI o1-tier logic at zero operational cost. If your goal is complete data sovereignty and blistering inference speed, pair Proxmox VE LXC containerization with AWQ quantization on vLLM.
People Also Ask
Can I run DeepSeek-R1 on dual RTX 3090 GPUs?
Yes. Dual RTX 3090 GPUs provide 48GB of total VRAM, which comfortably fits the DeepSeek-R1 Distill 70B model in Q4_K_M GGUF format or the Distill 32B model in uncompromised AWQ/FP8 format at over 50 tokens per second.
Is vLLM faster than Ollama for local LLM inference?
Yes. While Ollama (built on llama.cpp) is simpler for desktop use, vLLM utilizes PagedAttention, continuous batching, and chunked prefill, resulting in 2x to 3x higher throughput when handling concurrent requests or agentic API workflows.
Does quantizing DeepSeek models destroy their reasoning ability?
Testing confirms that modern 4-bit quantization methods (like AWQ and GGUF Q4_K_M) retain more than 97.5% of the full FP16 model’s mathematical and coding performance, making the slight precision loss imperceptible in daily programming tasks.