Multiple GPUs can split a model or host independent replicas, but VRAM is not always pooled like one larger card. Communication, PCIe lanes, topology, power, cooling, runtime support, and workload shape performance. Two cheap cards can cost more and perform worse than one suitable card after the whole system is counted.

// concept

Choose the Parallelism Goal

Model/tensor/pipeline splitting distributes one model so weights or computation span devices. It may enable a model that does not fit on one GPU, but cross-device communication adds cost.

Data parallel replicas load separate model copies and route different requests to each. This can improve throughput when the model fits on each device, but it does not combine VRAM for one larger request.

Independent workloads assign different models or queues per GPU. This is often simpler for a small office and can isolate interactive from batch work.

Framework terminology differs. Use the exact current documentation for llama.cpp, vLLM, or another runtime and verify whether mixed devices, unequal memory, peer access, and the chosen backend are supported.

// concept

Audit the Platform

Before buying a second card, document motherboard slots, physical spacing, electrical lane widths, CPU lane allocation, PSU continuous capacity/connectors, wall circuit and backup power, case clearance, airflow, device temperatures, and OS/runtime support.

A slot that physically fits may run fewer PCIe lanes or disable another device. Closely stacked cards can recirculate hot air. Different VRAM sizes and generations can make balanced splitting difficult. Consumer platforms may lack high-bandwidth interconnects used in data-center benchmarks.

// concept

Compare Three Architectures

Test or model:

one larger suitable GPU;
two lower-cost GPUs splitting one model;
two replicas or separate queues.

Include upfront components, PSU/case/board changes, energy/cooling, software time, failure domains, and resale/warranty. Benchmark quality, TTFT, throughput, p95, VRAM per device, bus traffic where observable, power, thermals, and recovery if one GPU/process fails.

// worked_example

Worked Example

Two used GPUs provide enough combined weight capacity, but the model split has slow p95 because the platform’s topology and runtime move data between cards. Two independent replicas of a smaller model deliver better throughput for the evaluated task. The larger model also fails to improve task quality enough to justify the latency.

The team chooses replicas with a health-aware router and accepts that each request is limited to one card’s model. It does not advertise combined VRAM as one pool.

// concept

Reliability Controls

Pin device identifiers carefully, expose per-GPU telemetry, and test restart/routing behavior. If one replica fails, stop routing new work to it. Model splitting may lose the whole service when one GPU fails, so document recovery and spare strategy.

Avoid installing unreviewed distributed/RPC services on open interfaces. llama.cpp’s RPC features carry explicit security considerations; keep unencrypted/unauthenticated components on trusted isolated networks only as documented, or add an approved secure boundary.

// failure_cases

Failure Cases

7 cases to diagnose

Adding VRAM numbers and assuming one address space.
Ignoring lane width, topology, or device communication.
Under-sizing PSU, connectors, case, and backup power.
Mixing unstable used cards without sustained testing.
Comparing multi-GPU speed on a different model/precision.
Exposing distributed worker ports without protection.
Assuming more GPUs always reduce latency.

// pakistan_angle

Pakistan Angle

Used multi-GPU builds can look attractive in PKR, but double the points of failure and complicate warranty. Gather dated quotes for the complete platform and qualified electrical review. Include heat management and load-shedding recovery.

If replacement cards are hard to source, standardized single-GPU nodes may be easier to repair than one custom multi-GPU tower. Evaluate local service availability and downtime, not just benchmark cost.

// hands_on

Hands-On Exercise

Design the three architectures above for one measured workload. Create a compatibility/topology diagram, full PKR cost model, power/thermal budget, benchmark plan, failure/recovery test, and security boundary. Recommend one or “do not buy yet.”

// completion_rubric

Completion Rubric

3 grading bands

Complete
parallelism goal, platform topology, full cost, measured performance, security, and failure behavior support the choice.
Needs revision
the design fits physically but communication, recovery, or total cost is not validated.
Not complete
combined VRAM, unsafe power, or internet-exposed worker ports are assumed acceptable.

// sources

Sources

3 official sources — check every claim yourself