CUDA is NVIDIA’s platform and programming model for using supported NVIDIA GPUs for general-purpose computing. An inference application usually sits above several layers: model/runtime, framework libraries, CUDA runtime/toolkit components, NVIDIA driver, operating system, and GPU hardware. Compatibility across those layers determines whether acceleration works.

// concept

Separate the Components

GPU driver: lets the operating system and applications communicate with the GPU; it includes the CUDA driver interface.
CUDA Toolkit: developer tools, headers, compiler, libraries, and samples used to build CUDA applications.
Framework/runtime package: PyTorch, llama.cpp, Ollama, vLLM, or another tool may ship or expect particular CUDA libraries.
Application/model: uses the framework/runtime and may add its own compute-capability or precision requirements.

Installing the newest full toolkit is not always necessary for a prebuilt application, and nvidia-smi showing a CUDA version does not necessarily mean that exact toolkit is installed. It reports driver capability information. Follow the application’s supported installation matrix.

// concept

Why the Driver Matters

The driver must support the GPU, operating system, and CUDA components used by the application. A driver that is too old for a runtime can cause initialization or library errors. A very new driver does not repair a framework built without the correct GPU architecture or backend.

NVIDIA documents compatibility rules and minimum driver requirements. Containers add another layer: the host driver remains critical even when CUDA user-space libraries are inside the container. Do not randomly mix installer methods or copy DLL/shared-library files between environments.

// concept

Inspect Before Installing

Capture a baseline:

// powershell2 lines

nvidia-smi
Get-CimInstance Win32_VideoController | Select-Object Name,DriverVersion

On Linux use the documented package/system tools. Record OS build, GPU, driver, application/runtime version, package environment, and exact error. Check the runtime’s official hardware and software requirements, then choose the supported installation route.

For PyTorch, verify the installed build and availability inside the exact environment running the application:

// python4 lines

import torch
print(torch.__version__)
print(torch.cuda.is_available())
print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else "no CUDA device")

This test says whether that PyTorch process sees CUDA; it does not validate every model operation.

// worked_example

Worked Example

An operator sees “CUDA 13.x” in nvidia-smi and installs several toolkit versions to make a prebuilt runtime work. The real issue is that the Python virtual environment contains a CPU-only framework wheel. The fix is to follow the framework’s official selector and reinstall in the correct environment, not keep changing the host driver.

Afterward, the operator runs a small tensor/device test and the actual model, checking GPU memory and output correctness. Configuration is recorded for rollback.

// failure_cases

Failure Cases

7 cases to diagnose

Confusing driver capability output with the installed toolkit.
Installing every CUDA version “just in case.”
Mixing Windows, WSL, container, and native paths mentally.
Using a framework command in a different virtual environment.
Updating a working production driver without rollback or maintenance window.
Treating GPU detection as end-to-end model validation.
Downloading drivers or libraries from unofficial mirrors.

// pakistan_angle

Pakistan Angle

Large driver/toolkit downloads may be slow, so save official installers or package metadata according to license and organizational policy, plus checksums where provided. Keep a tested rollback route; do not depend on finding the same package during an outage.

Used and older GPUs may lose support in newer stacks. Verify compute capability and current framework/runtime support before purchase. A card displaying output is not proof that the planned CUDA workload is supported.

// hands_on

Hands-On Exercise

Draw your machine’s stack from application to hardware. Record versions and sources, run a read-only driver check and framework availability test, then execute one small model operation. Explain what each result proves and does not prove. Make no system changes unless a documented incompatibility exists.

// completion_rubric

Completion Rubric

3 grading bands

Complete
driver, toolkit/runtime, framework, environment, and GPU are distinguished and verified with official compatibility evidence.
Needs revision
versions are listed but environment boundaries or proof of actual model execution is missing.
Not complete
components are changed randomly or downloaded from untrusted sources.

// sources

Sources

3 official sources — check every claim yourself