My Local AI Workflow: Claude, llama.cpp, and Cline on a Proxmox LXC

· 6 min read
local-aiself-hostinghomelabdeveloper-tools

The Setup

I’ve been running AI tools locally for a while now because I don’t feel right about sending private data to the cloud, value ownership and control over my data, and because there’s something deeply satisfying about watching a language model run on hardware you bought, in a container you configured, on a network you manage. That satisfaction compounds when you can pull it up from anywhere on your phone.

The short version: I run a llama.cpp LXC on my former gaming PC via Proxmox, access it from my laptop, iPad and phone over Tailscale, and use Claude for planning and heavy lifting while the local model handles quick tasks and experimentation.

The Local Proxmox AI Server: My Former Gaming PC

My local inference server is a repurposed gaming PC running Proxmox VE. It’s not a powerhouse by datacenter standards, but it’s more than capable of running quantized 7B models on CPU:

  • 7800X3D + 4070 TI Super — NVIDIA reigns supreme on x86. The 7800X3D’s 3D V-Cache gives it a nice boost for inference workloads, and the 4070 TI is a great GPU for gaming and general use, 16GB VRAM is enough for smaller models and experimentation.
  • 12 CPUs pinned to the Debian LXC, leaving me 4 cores for the Proxmox host
  • 24GB DDR5 RAM allocated to llama.cpp, which is enough for any VRAM spillage and gives me some breathing room for larger context windows
  • Local NVME storage I have two 2TB NVME drives running in a ZFS mirror for containers and models. It’s fast and redundant, and I have a backup strategy in place for the most important data.

Stability and uptime are good enough for my needs. The LXC restarts on failure, and I have monitoring set up to alert me if the server goes down. It’s not a production-grade setup, but it’s reliable for my personal use.

The Inference Server: llama.cpp

llama.cpp is the workhorse. It can run quantized GGUF models on CPU with surprisingly good throughput, and the built-in HTTP server (llama-server) speaks an OpenAI-compatible API, which means any tool that talks to OpenAI can point at it with a URL swap.

My typical install flow inside the container:

apt install -y build-essential cmake git
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_NATIVE=ON
cmake --build build --config Release -j$(nproc)

For model weights I pull Q3_K_L or Q4_K_M GGUF quantizations from Hugging Face — they hit a sweet spot between quality and memory footprint. I primarily run Qwen3.6-27B and Qwen3.6-35B-A3B for code tasks. Depending on the model and quantization, I can get anywhere from 30 to 90 tokens/second. It’s not blazing fast, but it’s perfectly usable for interactive coding sessions, brainstorming, and quick iterations.

Running the server:

./build/bin/llama-server \
    -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:IQ3_XXS \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --presence-penalty 1.5 \
    --min-p 0.00 \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --host 0.0.0.0 \
	--port 8080 \
    --no-mmproj \
    --ctx-size 128000

The Network: Tailscale

Tailscale is the glue. Its an overlay VPN network based on WireGuard. I have it installed on every node in my cluster and on my laptop and phone. The LXC container is a Tailscale node, which means llama-server is reachable at a stable 100.x.x.x address from anywhere I have Tailscale running — home, coffee shop, or traveling.

Install on Debian is a simple curl from the Tailscale website. Afterwards, I can point the server to listen on port 8080 and it becomes accessible over the Tailscale network with a readable name (llamacpp.x-x.ts.net) instead of an IP address. No need for dynamic DNS or port forwarding:

tailscale serve 8080 --bg

No port forwarding, no VPN config, no firewall rules to manage. It just works. I also run the Tailscale subnet router on my main Proxmox host so my laptop can reach the whole LXC network without needing Tailscale installed on every single container.

The IDE Layer: Cline

Cline is a VS Code extension that exposes an agentic coding assistant with explicit tool use — file reads, writes, shell commands, browser actions. It supports custom OpenAI-compatible endpoints, so pointing it at my llama.cpp server is just filling in the base URL:

https://llamacpp:8080/v1

For lighter tasks — autocomplete, quick explanations, boilerplate — the local model is fast enough and costs nothing to run. Latency from a 7B Q4 on a quad-core Tiny PC is around 8–15 tokens/sec on CPU, which is fine for interactive use.

For anything that requires deeper reasoning — complex refactors, architectural questions, long context — I switch Cline to the Claude API. The toggle is a dropdown. It takes two seconds.

Claude for the Heavy Lifting

I use Claude (via Anthropic’s API and Claude Code CLI) for tasks where the local model runs out of steam:

  • Large context windows — ingesting entire repos or multi-file diffs
  • Complex reasoning chains where a 7B model loses the thread
  • Code review with nuanced feedback, not just syntax fixes
  • Drafting and editing (like this post, somewhat recursively)

The division of labor feels natural after a few weeks. Local models are free, fast, and private for throwaway tasks and exploration. Claude is the better pair programmer when it matters.

What’s Working, What Isn’t

Works well:

  • Local inference for code generation, summarization, and quick Q&A
  • Full offline capability when Tailscale connectivity is unavailable
  • No per-token cost for the high-volume, low-stakes requests
  • Privacy — nothing leaves the LXC for local model requests

Rough edges:

  • CPU inference tops out around 15 t/s on my hardware.
  • Context length on local models is still constrained by RAM. 8192 tokens is workable; 32K requires trimming.
  • Model management is manual — keeping weights updated, tracking new quantization releases, disk space. It’s a small hobby tax.
  • llama.cpp’s server doesn’t support streaming tool use as well as the hosted APIs yet, which makes some Cline agentic flows a little janky with local models.

The Bigger Picture

There’s a version of this that’s just nerd infrastructure for its own sake. But I think there’s a real principle underneath it: the more capable and ubiquitous these models get, the more it matters that you can run them yourself. Not as a rejection of hosted AI — I pay for Claude and I think it’s worth it — but as a hedge and a complement.

The fact that I can SSH into a container from my phone, kick off a local inference job, and have it respond over my own network without a single request hitting an external API feels important. It’s the same instinct that drove me to self-host everything else: own the data, own the stack, stay in control of the thing.

The models will keep getting better. The hardware will keep getting cheaper. The setup I’m describing today will feel quaint in a year. That’s fine — the workflow adapts. The ownership principle stays the same.

All articles