Running a Private Local LLM as the Engine Behind My Homelab Automations

Most of my homelab automation is not scripts. It is loops: gather data, have a model reason about it, take an action. Email triage, log analysis, blog drafting, cron jobs that read telemetry and write analysis reports. All of them hit the same thing: one private, always-on, OpenAI-compatible endpoint.

That endpoint is not an API. It is llama-swap in front of llama.cpp, running on the same box that hosts the rest of my stack.

Why local in the first place

Three reasons, in rough order of importance.

First, data. I process email headers, network telemetry, log files, and personal content. I do not want that leaving the LAN, and I do not want to think about which parts I scrubbed before sending.

Second, cost and rate limits. Automation calls models in bursts: a cron job fans out to a few subagents, each making a dozen calls. A bill and rate limits for that shape of traffic is a real friction point. Local inference is free once the hardware is paid for.

Third, control. When I want a bigger context window, a different sampling preset, or a model swap mid-week, I edit a config file and reload. There is no provider to wait on.

What llama-swap is

llama-swap¹ is a thin proxy in front of llama.cpp’s² llama-server. It exposes one stable URL and manages a pool of model entries behind it. Each entry in llama-swap-config.yaml names a GGUF file, a port, and the server command to launch; the proxy starts and evicts backends on demand as requests arrive.

Three properties make this the right shape for automation: one stable URL that every client points at, so swapping models is a backend event and not a client change; config as data, with macros for the llama.cpp build path, context sizes, and KV quants, so adding a model means changing a file path; and a systemd user service rather than a root daemon, which gives the isolation I want for something that owns both GPUs.

One gotcha: the config resolves environment variables at load, and an unset PORT variable silently turns ${PORT} into an empty string, so the backend binds the wrong port or fails to start. The fix is boring but mandatory: export the variable in the context the service launches from.

The model behind the automation

I run Qwen3.8-27B, a dense hybrid-attention model, on a pair of RTX 5080s with a tensor split across the two cards. A few settings matter more than the model choice for automation:

• Context window. My agent framework enforces a 64K minimum for agentic workloads, and real sessions run much longer than that. I serve on the order of 230K context, with the KV cache quantized (q5_1 for K and V, matching quants on the draft layer) to fit in 32GB of combined VRAM. Unquantized, this does not load.
• Speculative decoding. The model ships MTP draft heads, and I run those alongside ngram speculation. On normal prose the server does roughly 71 tokens per second; on repetitive content like code and boilerplate it reaches 200 or more. That gap is why one endpoint can serve interactive chat and batch cron work.
• Sampling. I run explicit sampler chains per model instead of relying on server defaults, because defaults that are right for chat are wrong for structured automation output.

How the automation talks to it

The interface is the OpenAI chat completions API on loopback. My agent framework, Hermes, treats the local server exactly like any hosted provider: a model registry entry, a health check, a retry path. Cron jobs fire on schedule, make requests, and write results to disk or into my notes vault.

Model selection is where llama-swap earns its keep. I keep several entries: a small fast model for cheap classification, the 27B for main agentic work, and a narrative model for generation tasks. The small model handles things like “is this log line anomalous”, and the heavy model only loads when a task needs it. Loading a 20GB GGUF from NVMe takes seconds, not minutes.

Pitfalls that bit me

A few of these cost me real debugging time, and they all live in the gap between “the config says X” and “the process is doing X”:

1. Health check timeouts. The proxy health-checks each backend after a swap, and the default 30 seconds is shorter than the time a large model takes to load. The symptom is a 500 on the first request after a swap while the model itself is fine. The fix is a larger healthCheckTimeout.
2. Reasoning runaway. Thinking models will spend their entire reasoning and prediction budget even on a one-word prompt. A “say OK” probe hangs for minutes and returns megabytes. For non-reasoning tasks I explicitly disable thinking in the request payload.
3. Config edits evict the running model. Editing the config triggers a reload that drops the loaded backend and frees its VRAM, so a session’s prompt cache is gone. The subtler half: the new command registers, but the old process keeps serving until the next request forces a swap. Verify what is running by reading the process command line, not the config file.
4. Endpoint shape. The config stores a bare base URL. POSTing to the base instead of the completions path returns a 405 that looks like an auth bug but is a missing path segment. Normalize the URL in every client, and test with a small timed curl before blaming the model.

The tradeoffs I live with

The honest version: this is a 32-thread 9950X3D box with two 16GB GPUs, and the model occupies both of them while loaded. When I need the GPUs for something else, I restart the service and both cards are free in about four seconds; the next request reloads the model. That round trip is the price of sharing the machine, and for my workload it is cheap.

The other tradeoff is maintenance. A local inference stack is a build pipeline, a driver surface, and a config file that all interact. llama.cpp releases move fast, and new model architectures sometimes land in main a few days before tagged builds. My daily build job tracks tags, and when a new model fails on architecture, I confirm the upstream merge is in the tag before rebuilding anything.

None of this is glamorous. But the automation runs 24/7 on hardware I own, on data that never leaves the LAN, at a speed that has been adequate for everything I have thrown at it so far.