Every guide for this ends the same way: don’t hit update.
That is not a plan, it is a superstition. An unattended apt upgrade does not know it was told to be careful, and “never touch it again” leaves you on a kernel you cannot leave.
Four things have to agree: the kernel, its headers, the NVIDIA driver built against them, and CUDA. Break one and the ones after it go with it. A kernel bump invalidates the DKMS-built driver module, and CUDA goes down with the driver.
So I pin them. Not because updating is dangerous, but because they are coupled, and once you know what has to move together you can upgrade on purpose instead of hoping.
What this tutorial gives you
A clean Proxmox VE 9 host with an NVIDIA card passed into an unprivileged LXC, running llama.cpp on CUDA. Step by step, reproducible, with a verification command at every version-sensitive step so you can check the machine against the page instead of taking my word for it. The host driver comes from NVIDIA’s .run installer with --dkms, the container gets the same file again with --no-kernel-module so the userspace version matches by construction, and the CUDA toolkit comes from apt behind a fence that stops it overwriting the driver.
What you get at the end is a choice, and that is the part most guides leave out. You can stay exactly where you are for as long as you like, pinned, with apt unable to move the pieces out from under you. Or you can hit update on purpose, in the right order, knowing which four things have to move together. Both are supported, and there is a section for each. Five parts and an afternoon the first time.
If you came for the build and not the argument for it, skip to Building it. The reasoning is below, folded up, and nothing in the build depends on reading it.
Why bother, and what 16GB actually buys you The models that fit, the benchmarks behind the picks, and why the money argument is the weak one. None of it is needed to follow the build.
The word in that title is “now”, and the hardware is not what changed
16GB of VRAM has been ordinary for a while. What changed is that there is finally something worth putting on it.
Every step below is plumbing. Pin a kernel, match a driver, fence off apt, pass a device node into a container. None of it is interesting on its own, and none of it would be worth an afternoon if the only thing you could run at the end was a model three years behind whatever you get from an API.
That gap closed, and it closed for one specific reason: the labs publishing open weights got good. Not open source, which is a different and mostly unearned word here. Open weights. The learned parameters are downloadable, so the model runs on your hardware, offline, unmetered, and unchanged until you change it.
The models I actually run are Qwen3.6-27B and Qwen3.6-35B-A3B, both from Alibaba. Which quantization is not a detail at this size. It is the whole question, and picking the obvious one is how you find the ceiling: dense Q4_K_M does not fit, because it wants about 14.9 GiB before the KV cache exists at all, against roughly 15.6 GiB usable, and then runs out of room building it.
The 35B is a mixture of experts, which is the only reason a number that size belongs in a post about 16GB. Run it as UD-Q3_K_M with the experts offloaded and it holds roughly 3 to 6 GiB of VRAM out past 250k tokens of context, which is where I stopped testing rather than where it stopped working. That is the pick when the input is long, which for me it usually is: a feature-length video is nothing to it.
The 27B is dense, and dense is what fills the card. IQ3_M is the one that goes the distance: about 12.4 GiB to load, and it still sweeps out to 80k context at just over 15 GiB. IQ4_XS graded the same as it on my runs and caps out near 16k, so the lighter quant is not the compromise the numbering implies.
I measured this rather than guessing. The tooling is localai-16gb-bench, the write-up is FINDINGS.md, and the raw per-run sweeps sit under results/ in localai-16gb-bench-results. Every number in this section comes out of those CSVs.
I did not pick either out of ideology. I picked them because they are the ones that fit in the VRAM I have and they hold up on the work I actually give them. That work is not code. It is summarisation, home automation, and a growing bank of notes worth keeping searchable. Most of it runs through Open Notebook, an open-source implementation of NotebookLM, and that is the point rather than a detail. NotebookLM runs on Google’s machines. This one runs on mine: my hardware, my disk, whichever model I point it at. I own it. Google does not.
The loop is the point, not any one summary. A long video goes in. What comes back is not a paragraph restating it, it is insights, questions worth chasing, bullets, the analysis I would not have sat through the whole thing to write down myself. The parts worth keeping get saved as notes. Do that for a year and the bank is the artifact. Long inputs, run repeatedly, on material I would rather not hand to a hosted endpoint.
That is the whole “now”. It is not that the card got bigger. It is that the weights got good and stayed downloadable.
What this buys you, and what it does not
It is worth being precise, because the case for local is routinely made on the one ground where it is weakest.
Local is not cheaper per token
It is not close. Hosted inference is in a price war right now and the per-token rates have gone somewhere a card you paid for up front cannot follow. If someone tells you a home server pays for itself against current API pricing, ask to see the arithmetic.
What it is instead is fixed, and smaller than the ceiling makes it sound. The card idles at about 10 W and only reaches its 250 W cap while it is actually processing, which is not long: it gets through a video faster than the video plays. A job is a few minutes at 250 W and then back down to 10, not an hour of it. Both numbers are knowable before you start. A meter has neither.
There is more on the table there and I have not gone after it. I know from gaming that undervolting this card would make it run faster and cooler, not merely cheaper, which is the sort of free lunch that is hard to argue with. I have left it alone anyway. I would rather have stable than optimal on a box that is supposed to sit there and work.
The two sides are also moving in opposite directions. Per-token rates keep falling and hardware keeps going up, so the pays-for-itself arithmetic gets worse from both ends at once, and it was not close to begin with.
So buy accordingly, and the word that matters is used. Do not buy new hardware for this. New prices have been going up and nothing you run at home earns that back against hosted rates.
Do not overpay for used either. Do not run out and buy a used 3090 at 1300 just to do inference. That is the same mistake with a different sticker on it. A 4070 Ti Super is 16GB, and that class lands somewhere around 700 to 800. The number is in flux and you have to go looking, but they are findable. I paid 850 for mine.
Understand who the pricing is aimed at, too. Gamers used to be the demographic getting squeezed on cards. That is not who it is any more. It is local AI buyers, and the unified-memory machines that keep getting recommended into this conversation are now reaching three and four thousand dollars, which is viable and is not what this post is about. A 3090 is a fine card. Just have something to do with it besides inference, or get it for a steal, and do not go out of your way.
And be honest about which one you are, because it decides whether any of this is worth your weekend. This is for tinkering. It is for curiosity. It is for a card that is already in the machine because you game on it, where the inference is upside on hardware you already bought. Those are the good reasons, and they are enough. Saving money is not one of them, and if the arithmetic is what brought you here, the arithmetic says use an API.
What you get for the weekend is that you learn things you would not take on faith from someone else’s table. Two quants one step apart graded the same on my benchmarks, and the lower one left five times the context. The obvious quantization is the one that will not load at all. An abliterated build graded identically to the base it came from, so that modification was free on the answers I scored.
That last one also shows what a small benchmark cannot tell you. Free on coding quality says nothing about what else abliteration changed, and my one security prompt failed to discriminate: every model I ran, modified or not, answered it the same way. Knowing which of your questions did not work is part of the return too.
None of this is on a spec sheet, and none of it is transferable: it is true of this card, at this size, with these weights.
What you get instead is a different set of properties, and they are the ones that actually made me build this:
- It is not metered. There is no per-token cost, so there is no reason not to leave something running, or to throw a three hour video at it, or to run the same job forty times while you get a prompt right.
- Nothing leaves the machine. For anything under an NDA, or anything personal, that is not a preference. It is the requirement. There is zero chance I hand my health records to something I do not own, short of GDPR terms or a written zero-retention guarantee, and both of those are still someone else’s promise about what they do with a copy they already have. Weights on my own disk are the version where there is no copy to promise about.
- It does not change underneath you. The weights on my disk are the weights I benchmarked. A hosted model can be updated, deprecated, or re-tuned between one Tuesday and the next, and your evaluation goes with it.
- It does not need an account that can go away. Billing failures, suspensions, and rate limits are all real and none of them apply to a file.
- You decide what it will answer. A hosted model’s idea of a reasonable question is not negotiable and not appealable, and it gets it wrong in both directions. Security research on my own VLAN is the obvious case: ordinary work that reads to a filter like the thing it was trained to refuse. I want to ask my own machine anything and not be locked out of my own work by a refusal I have no way to argue with. On weights I hold, what it will answer is a setting rather than someone else’s policy. I ask, and I get an answer.
That last one deserves saying properly, because the failure is not one bad call. It is cumulative. Each question is ordinary on its own and the trajectory is what gets scored, so the refusal lands on the person doing the most careful work. Ask one thing and you get an answer. Enumerate the whole surface, which is what a threat model is, and you start to look like reconnaissance. That inverts the thing the filter is for: the careless get served and the thorough get stonewalled.
It is also aimed at a moving target. Legality is local. Weed is legal in New York, Washington and Oregon and is not legal in plenty of places I could be sitting when I ask, so one hosted policy has to either pick somebody’s jurisdiction or take the intersection of all of them and refuse on behalf of the strictest. Ethics moves far less than that. Jaywalking is illegal and invented, which is most of what you need to know about treating the two as the same axis.
Those are the reasons. The price argument is the weak one, and I would rather say so than have it pointed out.
Dated claim, 2026-08-03. The open-weight landscape is moving fast enough that anything specific I say about which model to run has a short half-life. Alibaba, DeepSeek, and Moonshot AI have all shipped in recent months, and the sizes and the licences keep moving. Treat every model name in this post as an example of the category, not a recommendation with a shelf life.
Building it
This is a clean Proxmox VE 9 install to a working CUDA container: llama.cpp inside an LXC, GPU passed through, driver installed from NVIDIA’s .run on both sides. The internet is full of guides for it, and most are subtly wrong, outdated, or skip the one step that will actually break your setup three weeks later. I built this one by hand, no LLM. It took a while.
Credit where it’s due: the backbone of this workflow comes from Digital Spaceport’s excellent Proxmox AI homelab series, specifically Step 1: Ollama + OpenWebUI on a Proxmox LXC and Step 2: llama.cpp on a Proxmox 9 LXC. The video for Step 1 is Local AI Server Setup Guides Proxmox 9: OpenWebUI + Ollama in LXC with GPU Passthrough. If you want video walkthroughs and a beginner-friendly on-ramp (including the Proxmox install itself), start there. What follows is my take on the same host-.run-plus-container-.run approach, with changes and additions from my own setup. I’ll call out where and why I diverge.
Here’s the shape of it:
| Layer | How |
|---|---|
| Host driver + kernel module | .run installer with --dkms |
| Container driver userspace | the same .run file with --no-kernel-module |
| Container CUDA toolkit | apt, fenced off so it can’t wreck the driver |
Why the .run installer instead of apt packages? One reason above all: an LXC container shares the host’s kernel, so it only needs the driver’s userspace libraries, and those must match the host’s driver version exactly, or you get the dreaded Failed to initialize NVML: Driver/library version mismatch. When both host and container install from the literal same file, that match is guaranteed by construction. No version archaeology, no “which repo published what.”
The honest trade-offs: apt has no idea your driver exists (which cuts both ways; we’ll handle it), and kernel changes require a little more care. I think it’s worth it. If you disagree, there’s an appendix for you.
A note on versions. I wrote this against Proxmox VE 9.x, driver 595.84, CUDA toolkit 13.3. By the time you read it, those numbers are probably stale. Every version-sensitive step has a verification command. Use them; don’t trust me.
Where you should be starting from
- A fresh Proxmox VE 9 install, SSH access as root
- GPU physically in the box and showing up in
lspci | grep -i nvidia - Booted into the kernel you plan to keep. If you want the latest, do
apt full-upgradeand reboot now, before anything else, because we’re about to pin whatever’s running. - Nothing NVIDIA-related installed yet. Check:
dpkg -l | grep -Ei 'nvidia|cuda' || echo "clean"
lsmod | grep nvidia || echo "no nvidia modules loaded"If those come back non-empty, purge the old attempt before continuing. Half-installed driver remnants are where debugging sessions go to die.
Part 1 is all on the host. Parts 2–5 involve the container.
The things that will actually bite you
I’m putting these up front because every one of them cost me (or someone on the Proxmox forums) real time:
- Pin the kernel first. A casual
apt upgradethat pulls a new kernel leaves your driver module built for the old one. With the.runmethod this is worse than usual, because apt doesn’t even know there’s a driver to protect. - Headers must match the running kernel. If they don’t, DKMS quietly builds nothing,
nvidia-smidies, and/dev/nvidia*never appears. This is the #1 failure mode, full stop. Also: the package isproxmox-headers-<version>now, not the oldpve-headers-*. Plenty of older guides will steer you wrong here. nvidia-smiworking does not mean CUDA works. If/dev/nvidia-uvmis missing, the driver looks fine but compute fails. Andnvidia-persistencedloads thenvidiamodule but not reliablynvidia_uvm. We fix this properly in 1.6.- The no-kernel-module flag’s exact name varies by installer version. Some installers take
--no-kernel-module(singular), others--no-kernel-modules(plural). Digital Spaceport’s guides use the plural, other docs the singular. Don’t guess; ask your installer:./NV*.run --help | grep -i kernel-moduleand use whatever it lists. Get it wrong and the installer rejects the flag outright, which is at least a loud failure. /dev/nvidia-caps/may or may not exist on your card (it’s a MIG thing). Bind device nodes that don’t exist and your container simply won’t start. Look before you bind.- The CUDA repo will happily clobber your
.runuserspace. You need the repo in the container for the toolkit, so we put up a fence (an apt pin) before enabling it. - Keep the
.runfile. It’s your version pin, your container installer, and your recovery tool, all in one ~300 MB blob.
CUDA 13.2 produces gibberish. Use 13.3.
If your model loads, offloads to the GPU, reports no errors, and returns nonsense, you’re probably on CUDA 13.2. It isn’t your quant and it isn’t your GGUF.
13.2 is the only version that does this. 13.3 fixed it, and 13.0, 12.8, and 13.1 are fine. The failure presents as a model problem rather than a toolkit problem, which is why it costs people days.
This is also the reason a version pin isn’t just caution. A bad release sandwiched between two good ones beats “don’t hit update”, which strands you on it, and beats “stay current”, which walks you into it on the way past.
Someone filed it and Unsloth noted it, across several model threads, an issue on unslothai/unsloth, and upstream in llama.cpp. For Qwen3.6-35B-A3B specifically, which is what I run.
The published fix is to drop below CUDA 13.0 or use Unsloth Studio. Both work. I pinned CUDA and kept running llama.cpp directly, because that’s the thing I wanted to run.
Where this diverges from the Digital Spaceport guides
So you’re not confused cross-referencing, here’s what I do differently and why:
The first three are links in the chain from the opening, in order:
- The kernel is pinned (1.1). It’s the first link, so it’s the first thing to hold still.
- Headers package. I use
proxmox-headers-$(uname -r). On Proxmox 9 that’s the current package family, and the versionedpve-headers-*names from older guides may not exist for newer kernels. The headers have to agree with the kernel you’re actually running or DKMS builds nothing, silently. - The host stays driver-only. Their guides install
cuda-toolkiton the host as well as the container. The container needs the toolkit; the host needs a driver. Less surface for anapt upgradeto reach.
The rest:
- apt is fenced off the NVIDIA packages (1.5). The driver came from a
.runfile, so apt was never managing it. The job is keeping apt from deciding otherwise. - A boot unit for
nvidia-uvm(1.6). The device node not existing is a failure you hit at boot, and then hit again the next morning. - No GL/X11 build dependencies. Their dependency line pulls in
freeglut3-dev,libx11-dev,libglu1-mesa-dev,libglfw3-dev, and friends. Those exist to build CUDA’s graphical samples. A headless inference box never renders anything, so I skip them. - Passthrough as config, not just clicks. Their guides wire up the GPU through the web GUI. That works, but I also show the raw
dev0:–dev5:lines for/etc/pve/lxc/<CTID>.conf. They’re copy-pasteable, they’re scriptable if you ever automate container creation, and they make thenvidia-capssituation explicit: you can see exactly which lines to leave out when/dev/nvidia-caps/doesn’t exist on your card, instead of discovering it when the container won’t start.
None of this is a knock on their guides. They’re aimed at getting a beginner to a working Ollama/llama.cpp box, and they do that well. This one adds the pinning and fencing that keeps the setup working after six months of unattended apt upgrades.
Alright. Let’s build it.
Part 1: Host driver via .run + DKMS
1.1 Pin the kernel
Whatever kernel you’re on right now is the one everything else in this guide gets built against. Lock it down:
uname -r # note it: this is your kernel now
proxmox-boot-tool kernel pin "$(uname -r)"
proxmox-boot-tool kernel list # confirm "Pinned kernel"
apt-mark hold "proxmox-kernel-$(uname -r)"1.2 Headers and build tools
apt update
apt install -y build-essential dkms pciutils proxmox-headers-$(uname -r)
# Don't just trust the install. Confirm the header tree is really there:
ls -d /usr/src/linux-headers-$(uname -r) && echo "headers OK" || echo "headers MISSING"You’ll see other guides pull in freeglut3-dev, libx11-dev, and a pile of GL packages. Those are for building CUDA’s graphical samples. A headless inference box doesn’t need any of it. Skip.
1.3 Get nouveau out of the way
The open-source nouveau driver grabs the GPU at boot before NVIDIA’s driver can. If lsmod | grep nouveau is already empty, skip ahead. Otherwise:
cat > /etc/modprobe.d/blacklist-nouveau.conf <<'EOF'
blacklist nouveau
options nouveau modeset=0
EOF
update-initramfs -u
reboot
# after reboot:
lsmod | grep nouveau # silence = success1.4 The actual driver install
Grab your driver from nvidia.com/drivers/unix. Production branch unless you have a specific reason to chase the new-feature branch; CUDA 13.x only needs r580 or newer, so anything current clears the bar.
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/595.84/NVIDIA-Linux-x86_64-595.84.run
chmod +x NVIDIA-Linux-x86_64-595.84.run
# Installers change flags between versions. Double-check yours:
./NVIDIA-Linux-x86_64-595.84.run --help | grep -i kernel-module
# --dkms means the module rebuilds itself when the kernel changes. You want this.
./NVIDIA-Linux-x86_64-595.84.run --dkmsThe installer asks a few questions:
- Open (MIT) or proprietary kernel modules? Proprietary for Ada (RTX 40-series) and older. That’s what the Digital Spaceport guide recommends and what this box runs. Blackwell (50-series) requires the MIT/open modules: same download, different option at this prompt. (NVIDIA has been nudging Turing-and-newer toward the open modules lately, and they do work on Ada if you prefer them. Just make the same choice on every future install.)
- Update X configuration? No. There’s no X here.
- 32-bit compatibility libraries? Not needed for compute.
One quirk worth knowing (Digital Spaceport documents this too): on some systems the installer complains about modprobe/headers and demands a reboot mid-install. If it does, let it reboot, then run ./NV*.run --dkms again and let it finish the second time. Annoying but normal.
Now, and this matters, archive the file. It’s your version pin and the exact installer you’ll push into the container later:
mkdir -p /root/nvidia-drivers
cp NVIDIA-Linux-x86_64-595.84.run /root/nvidia-drivers/Reboot and check your work:
nvidia-smi
dkms status # must show nvidia/<ver> built for your RUNNING kernel
ls -la /dev/nvidia*If nvidia-smi fails here, it’s almost certainly gotcha #2: the module never got built for the kernel you’re running. Confirm the headers from 1.2 exist, then build explicitly:
dkms install nvidia/595.84 -k $(uname -r)
modprobe nvidia && modprobe nvidia_uvmIf dkms status shows nothing registered at all, just re-run the .run installer.
1.5 Build a fence around it
Here’s the awkward part of a .run install: apt doesn’t know your driver exists. That means apt also won’t protect it. Some future package could pull in libnvidia-* as a dependency and stomp all over your carefully matched userspace. So we make those packages uninstallable:
# /etc/apt/preferences.d/nvidia-run-pin
Package: nvidia-driver nvidia-driver-cuda nvidia-kernel-dkms nvidia-kernel-common nvidia-kernel-source* cuda-drivers cuda-drivers-* libnvidia-* libcuda1
Pin: release *
Pin-Priority: -1Pin-Priority: -1 is apt for “never, under any circumstances.” Verify the coast is clear:
apt-get -s upgrade | grep -iE 'nvidia|cuda' || echo "nothing GPU-related will upgrade"1.6 The nvidia-uvm problem
This one is sneaky enough to deserve its own section. The persistence daemon loads the nvidia module and creates /dev/nvidia0 and /dev/nvidiactl at boot, but nvidia_uvm, the module CUDA actually computes through, doesn’t reliably come along. The result is a machine where nvidia-smi is green and happy while every CUDA program fails to initialize. In a container that autostarts at boot, this means your inference server is dead every morning until you SSH in and modprobe by hand.
Fix it twice: declare the modules, and add a systemd unit that creates the device node before Proxmox starts any guests:
cat > /etc/modules-load.d/nvidia.conf <<'EOF'
nvidia
nvidia_uvm
EOF
cat > /etc/systemd/system/nvidia-uvm-init.service <<'EOF'
[Unit]
Description=Create NVIDIA UVM device nodes before containers start
After=nvidia-persistenced.service
Before=pve-guests.service
Wants=nvidia-persistenced.service
[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-modprobe -c0 -u
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable nvidia-uvm-init.service
systemctl enable --now nvidia-persistencedThe only test that counts is a reboot. Reboot, touch nothing, and check:
lsmod | grep -E '^nvidia' # nvidia AND nvidia_uvm
ls -la /dev/nvidia-uvm # node exists
systemctl status nvidia-uvm-init.service --no-pager | head -61.7 Optional: persistence mode and a power cap
If you want a power limit (quieter, cooler, barely slower for inference), set it at boot with a cron line in root’s crontab, same approach as the source guides:
crontab -e
# add this line:
@reboot nvidia-smi -pm 1 && nvidia-smi -pl 250Apply it now for the current session too, and verify:
nvidia-smi -pm 1 && nvidia-smi -pl 250
nvidia-smi --query-gpu=power.limit,power.max_limit --format=csvPick your wattage deliberately against your card’s board power, or skip the -pl part entirely to run stock. One thing to remember: this line re-applies at every boot, so if you later change the limit at runtime and wonder why it reverted after a reboot: it’s this line. Change the value here, at the source.
Part 2: Telling the container about the GPU
Proxmox 9 passes devices into LXCs with devN: lines. The critical habit: bind only nodes that actually exist on your host. List them first:
ls -la /dev/nvidia*
ls -la /dev/nvidia-caps/ 2>/dev/null # exists on some cards, not othersThen edit /etc/pve/lxc/<CTID>.conf:
dev0: /dev/nvidia0
dev1: /dev/nvidiactl
dev2: /dev/nvidia-uvm
dev3: /dev/nvidia-uvm-tools
dev4: /dev/nvidia-caps/nvidia-cap1 # ONLY if the caps dir exists on your host
dev5: /dev/nvidia-caps/nvidia-cap2 # ONLY if the caps dir exists on your hostBind a caps node that isn’t there and the container refuses to start. That’s the whole reason for the look-first rule. Prefer clicking? Container → Resources → Add → Device Passthrough does the same thing.
Part 3: Container userspace, the payoff
This is where installing from a .run file earns its keep. The container shares the host kernel, so it needs only the driver’s userspace libraries, at exactly the host’s version. We have a file that is exactly the host’s version. Push it in:
# From the HOST:
pct push <CTID> /root/nvidia-drivers/NVIDIA-Linux-x86_64-595.84.run /root/NVIDIA-Linux-x86_64-595.84.run
pct enter <CTID>
# INSIDE the container:
apt update && apt install -y kmod pciutils
chmod +x /root/NVIDIA-Linux-x86_64-595.84.run
# Confirm the exact flag name for YOUR installer version first (see gotcha #4):
/root/NVIDIA-Linux-x86_64-595.84.run --help | grep -i kernel-module
/root/NVIDIA-Linux-x86_64-595.84.run --no-kernel-modules # or singular, per the check aboveThis install is quick. With no kernel module to build, you can accept the defaults it presents (proprietary, same as the host) and say no to the X config question.
Then the verification that this whole guide orbits around:
# INSIDE the container:
nvidia-smi # your GPU, not an NVML error
cat /proc/driver/nvidia/version # must equal the host's version, character for characterIf you get the NVML mismatch anyway, the versions diverged somewhere. Reinstall the container userspace from the file that matches what the host is actually running.
Part 4: CUDA toolkit in the container
The toolkit comes from NVIDIA’s apt repo. The same repo also ships driver packages that would love to overwrite the userspace you just installed. Fence first, repo second, same pin as the host:
# INSIDE the container: /etc/apt/preferences.d/nvidia-run-pin
Package: nvidia-driver nvidia-driver-cuda nvidia-kernel-dkms nvidia-kernel-common nvidia-kernel-source* cuda-drivers cuda-drivers-* libnvidia-* libcuda1
Pin: release *
Pin-Priority: -1Check what OS the container runs. The repo URL depends on it, and your container isn’t necessarily the same OS as your host:
grep -E 'PRETTY_NAME|VERSION_CODENAME' /etc/os-releaseFor Debian 13:
wget https://developer.download.nvidia.com/compute/cuda/repos/debian13/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
# Test the fence before trusting it:
apt-get install --dry-run cuda-toolkit-13-3 # must NOT pull nvidia-driver-* or libnvidia-*
apt install -y cuda-toolkit-13-3Note the package name: cuda-toolkit-13-3, versioned, not the floating cuda metapackage. The versioned name is toolkit-only and stays on its minor version forever, with no surprise jump to 13.4 during some unrelated upgrade. This is also where you avoid 13.2: the version you name here is the version you get. Wire up the path:
echo 'export PATH=/usr/local/cuda-13.3/bin${PATH:+:${PATH}}' >> ~/.bashrc
source ~/.bashrc
nvcc --versionIf you’re ever tempted by cuda-drivers or plain cuda in the container: don’t. With the pin in place they’ll refuse to install anyway, which is the pin doing its job.
Part 5: Prove the whole thing works
Say it with me one more time: nvidia-smi proves the driver, not CUDA. Test both:
# INSIDE the container:
nvidia-smi
nvcc --version
# Then a real compute test: deviceQuery from cuda-samples, or better,
# just launch your actual workload (llama.cpp etc.) and watch it
# offload layers to the GPU and claim VRAM.Driver fine but CUDA won’t initialize? That’s /dev/nvidia-uvm missing again. Back to 1.6.
When you eventually upgrade the driver
Nothing in this setup moves on its own. That’s deliberate. When you decide it’s time, the upgrade is pleasingly symmetric:
# HOST:
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/<NEW_VER>/NVIDIA-Linux-x86_64-<NEW_VER>.run
chmod +x NVIDIA-Linux-x86_64-<NEW_VER>.run
./NVIDIA-Linux-x86_64-<NEW_VER>.run --dkms # it uninstalls the old version itself
cp NVIDIA-Linux-x86_64-<NEW_VER>.run /root/nvidia-drivers/
reboot
nvidia-smi && dkms status # host healthy BEFORE touching the containerThen push the same new file into the container and repeat Part 3. Host and container move together, always, because they install from the same file. Toolkit bumps are independent: apt install cuda-toolkit-13-4 in the container, drop the old versioned package if you like, update the PATH line.
One piece of hard-won advice: never fix and upgrade in the same session. If the GPU is broken right now, get the current version healthy first, verify, and only then upgrade. Change three things at once on a down box and you’ll have no idea which one saved you, or which one made it worse.
If the kernel ever changes
The --dkms flag means the module should rebuild automatically on a new kernel. “Should” is doing some work in that sentence: a silent rebuild failure is the classic way this setup dies months later. After any kernel move:
apt install -y proxmox-headers-$(uname -r) # headers for the NEW kernel, first
dkms status # is nvidia/<ver> listed for $(uname -r)?
dkms install nvidia/595.84 -k $(uname -r) # if not, build it by hand
nvidia-smiIf DKMS flat-out can’t build (say, a kernel newer than the driver supports), re-run the .run installer, or put the kernel pin back and stay where you were. There’s no prize for running the newest kernel on an inference box.
Appendix: prefer apt on the host?
Fair enough. Install nvidia-driver-cuda + nvidia-kernel-dkms from the CUDA repo and apt-mark hold them. The container side of this guide doesn’t change at all: it’s still .run --no-kernel-module, just with a file you download to match whatever version the repo gave you, instead of the same file by construction. You gain apt tracking and ordinary upgrade transactions; you lose the guaranteed match and take on a small version-bookkeeping chore. Reasonable people choose either.
Consumer NVIDIA hardware plus Linux plus virtualization is a moving target, and this guide has a half-life. Re-check versions, URLs, and repo paths before pasting anything. And keep that .run file. Future you will thank present you.
The models have a shorter half-life than the plumbing does. The pinning, the fencing, and the .run symmetry will still be right long after the specific weights on my disk are the wrong ones to be running. That is the actual argument for building it this way: the box outlives whatever is currently the best thing to put on it.