Multiple GPUs can split a model or host independent replicas, but VRAM is not always pooled like one larger card. Communication, PCIe lanes, topology, power, cooling, runtime support, and workload shape performance. Two cheap cards can cost more and perform worse than one suitable card after the whole system is counted.
Choose the Parallelism Goal
Model/tensor/pipeline splitting distributes one model so weights or computation span devices. It may enable a model that does not fit on one GPU, but cross-device communication adds cost.
Data parallel replicas load separate model copies and route different requests to each. This can improve throughput when the model fits on each device, but it does not combine VRAM for one larger request.
Independent workloads assign different models or queues per GPU. This is often simpler for a small office and can isolate interactive from batch work.
Framework terminology differs. Use the exact current documentation for llama.cpp, vLLM, or another runtime and verify whether mixed devices, unequal memory, peer access, and the chosen backend are supported.
Audit the Platform
Before buying a second card, document motherboard slots, physical spacing, electrical lane widths, CPU lane allocation, PSU continuous capacity/connectors, wall circuit and backup power, case clearance, airflow, device temperatures, and OS/runtime support.
A slot that physically fits may run fewer PCIe lanes or disable another device. Closely stacked cards can recirculate hot air. Different VRAM sizes and generations can make balanced splitting difficult. Consumer platforms may lack high-bandwidth interconnects used in data-center benchmarks.
Compare Three Architectures
Test or model:
- one larger suitable GPU;
- two lower-cost GPUs splitting one model;
- two replicas or separate queues.
Include upfront components, PSU/case/board changes, energy/cooling, software time, failure domains, and resale/warranty. Benchmark quality, TTFT, throughput, p95, VRAM per device, bus traffic where observable, power, thermals, and recovery if one GPU/process fails.
Worked Example
Two used GPUs provide enough combined weight capacity, but the model split has slow p95 because the platform’s topology and runtime move data between cards. Two independent replicas of a smaller model deliver better throughput for the evaluated task. The larger model also fails to improve task quality enough to justify the latency.
The team chooses replicas with a health-aware router and accepts that each request is limited to one card’s model. It does not advertise combined VRAM as one pool.
Reliability Controls
Pin device identifiers carefully, expose per-GPU telemetry, and test restart/routing behavior. If one replica fails, stop routing new work to it. Model splitting may lose the whole service when one GPU fails, so document recovery and spare strategy.
Avoid installing unreviewed distributed/RPC services on open interfaces. llama.cpp’s RPC features carry explicit security considerations; keep unencrypted/unauthenticated components on trusted isolated networks only as documented, or add an approved secure boundary.
Failure Cases
- Adding VRAM numbers and assuming one address space.
- Ignoring lane width, topology, or device communication.
- Under-sizing PSU, connectors, case, and backup power.
- Mixing unstable used cards without sustained testing.
- Comparing multi-GPU speed on a different model/precision.
- Exposing distributed worker ports without protection.
- Assuming more GPUs always reduce latency.
🇵🇰 Pakistan Angle
Used multi-GPU builds can look attractive in PKR, but double the points of failure and complicate warranty. Gather dated quotes for the complete platform and qualified electrical review. Include heat management and load-shedding recovery.
If replacement cards are hard to source, standardized single-GPU nodes may be easier to repair than one custom multi-GPU tower. Evaluate local service availability and downtime, not just benchmark cost.
Hands-On Exercise
Design the three architectures above for one measured workload. Create a compatibility/topology diagram, full PKR cost model, power/thermal budget, benchmark plan, failure/recovery test, and security boundary. Recommend one or “do not buy yet.”
Completion Rubric
- Complete: parallelism goal, platform topology, full cost, measured performance, security, and failure behavior support the choice.
- Needs revision: the design fits physically but communication, recovery, or total cost is not validated.
- Not complete: combined VRAM, unsafe power, or internet-exposed worker ports are assumed acceptable.
Sources
- llama.cpp multi-GPU documentation
- llama.cpp server device-split options
- vLLM parallelism and scaling
Key takeaway: Multi-GPU only pays when the chosen parallelism matches the workload and the full topology, power, software, and failure costs are measured.