Troubleshoot CUDA conflicts by preserving the error and narrowing the layer. Reinstalling drivers, toolkits, Python packages, and Windows features at once destroys evidence and can turn one mismatch into several. Begin with read-only inventory and one minimal reproduction.
Capture the Failure
Save the full command, stderr/traceback, timestamp, application version, environment name, recent changes, and whether it ever worked. Record:
- OS/build and native, WSL, VM, or container boundary;
- GPU and driver from
nvidia-smi; - runtime/framework build and CUDA availability;
- Python executable and package locations where relevant;
- model/backend/precision and smallest failing operation;
- GPU processes, memory, and system logs.
Redact usernames, tokens, repository secrets, and personal paths before sharing logs.
Use an Error Taxonomy
Driver/device unavailable: the process cannot see a compatible device. Check host driver, device assignment, permissions, container GPU configuration, and environment boundary.
Library not found or wrong version: a required DLL/shared object is missing or another copy is found first. Inspect the documented environment and package manager; do not download individual DLLs from random websites.
Framework reports CUDA false: confirm you are running the expected interpreter/build. A CPU-only wheel or wrong virtual environment is common.
Unsupported architecture/kernel: the binary or operation may not support the GPU’s compute capability/data type. Check the framework/runtime support matrix and build flags.
Out of memory: first reduce model/context/concurrency and inspect other processes. OOM is not automatically a driver conflict.
Illegal memory access or device-side error: reduce to a minimal case, restart the affected process cleanly, preserve logs, and consult the owning project. Avoid continuing production traffic through a corrupted process state.
Narrow With a Ladder
- Does the OS/driver see the GPU?
- Does the exact runtime environment see it?
- Does a tiny supported compute operation pass?
- Does the smallest supported model pass?
- Which model size, precision, context, or feature triggers failure?
At each step, compare official requirements. Change one item, document it, retest, and keep a rollback. Prefer the package method recommended for that OS/framework. Mixing distribution packages, standalone installers, Conda, pip, and copied libraries can cause ambiguous resolution.
Worked Example
A Windows host runs Ollama successfully, while a Python script in WSL reports no CUDA device. That is not contradictory: they are different environments and runtimes. The operator checks WSL GPU prerequisites and the Python wheel in WSL rather than reinstalling the Windows application.
Another application fails only at long context with an OOM. Reducing context makes it pass. The driver is functioning; the configuration exceeds memory.
Failure Cases
- Removing a working driver before recording its version.
- Copying CUDA libraries into application folders.
- Assuming all errors containing “CUDA” need a toolkit install.
- Mixing environments and package managers.
- Ignoring release notes, compute capability, or backend support.
- Running cleanup commands from unverified forum posts.
- Sharing logs that expose tokens or user data.
🇵🇰 Pakistan Angle
Keep a small offline incident bundle: system inventory, approved installer/package references, hashes where available, last-known-good configuration, and rollback notes. This reduces dependence on stable internet during repair.
For a client machine, obtain permission and schedule downtime before driver changes. Clarify who owns warranty support. Do not flash firmware, open hardware, or bypass vendor controls unless authorized and qualified.
Hands-On Exercise
Take one real or simulated failure and create an incident note. Include exact error, sanitized inventory, taxonomy, five-step ladder, one-variable experiment log, result, rollback, and prevention. If no failure exists, create a wrong-environment PyTorch scenario without altering production.
Completion Rubric
- Complete: the failing layer is isolated with a minimal test, official compatibility evidence, sanitized logs, and reversible change.
- Needs revision: the issue is fixed but the actual cause or reproducible evidence is uncertain.
- Not complete: random driver/library changes, untrusted downloads, or destructive fixes are used.
Sources
- NVIDIA CUDA compatibility documentation
- NVIDIA System Management Interface
- PyTorch installation and verification
- Ollama troubleshooting
Key takeaway: Preserve evidence, identify the environment boundary, and climb from driver to minimal model one layer at a time.