Self-host an LLM on a VPS.
No GPU. No API key. No prompt ever leaves the building.

The honest version of this section is worth more than an enthusiastic one, because the fastest way to abandon a self-hosted model is to point it at the one job it is worst at and conclude the whole idea was silly.

Fits comfortably. Anything where the output is short and the value is in the judgement rather than the prose. Classifying a message into one of a handful of buckets. Deciding whether a support ticket is urgent. Extracting five fields from an invoice as JSON. Summarising a thread into three sentences. Tagging a document. Rewriting a product description. Translating a short string. Detecting whether two records describe the same person. Redacting names from a paragraph before it goes anywhere else. Every one of those consumes a long input and produces a short output, which is precisely the asymmetry a CPU handles well.

Fits, with patience. Work with no human waiting on it. Overnight batches, queue workers, a nightly pass over the day's documents, a scheduled report. If nothing blocks on the answer, a token rate that would be intolerable in a chat window stops mattering at all — a job that runs for six minutes at three in the morning is simply a job that ran.

Does not fit. An interactive assistant that people will actually enjoy using: at these rates the cursor crawls and the experience is worse than no assistant. Long-form generation — write me two thousand words — where the entire output is the slow part. Coding agents that iterate over a large repository, where both the context and the quality bar exceed what a small model can hold. Anything needing a hundred-thousand-token context, where the cache alone outgrows the machine. And image or video models, which are a different discipline entirely and genuinely do want a GPU.

If your workload lives in that last group, the sane answer is not to buy a bigger CPU box — it is a hybrid. Keep a local model for the bulk of the calls and route the handful that need frontier reasoning to a hosted API, which is exactly the shape the guide on running an AI agent 24/7 describes for the runtime side. That guide deliberately left the model itself out of scope. This one is the missing half.

Harden the machine first — key-only SSH, a firewall that allows nothing you did not ask for, unattended security upgrades. The first-hour checklist takes about an hour and this is a box that will shortly be holding every question you ask it.

Then install the runner. Ollama is llama.cpp with a model registry, a REST API and a systemd unit wrapped around it, and the one-line installer sets up all three:

apt update && apt install -y curl
curl -fsSL https://ollama.com/install.sh | sh

systemctl status ollama --no-pager
ss -ltnp | grep 11434
# → 127.0.0.1:11434 — loopback only. Leave it that way.

That last line is the one to read twice. Out of the box the service listens on the loopback interface, which is exactly right, and the most common mistake made in the next ten minutes is to set OLLAMA_HOST to 0.0.0.0 because something on another machine could not reach it. Step 04 is how to reach it properly; there is no version of this where port 11434 faces the internet directly.

Two settings are worth putting in place now rather than discovering later. Models are stored under /usr/share/ollama by default and they are large, so point that at wherever you have room. And the default keeps a model resident in RAM for five minutes after the last request, which is generous on a box that is also doing other work:

systemctl edit ollama

[Service]
Environment="OLLAMA_MODELS=/var/lib/ollama/models"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MAX_LOADED_MODELS=1"

systemctl daemon-reload && systemctl restart ollama

Parallel requests and multiple loaded models both multiply memory use, and on a machine sized to hold exactly one model there is no spare gigabyte for a second copy. Serialising the queue is not a limitation on this hardware; it is the only configuration that does not fall over.

This is the part of the guide that stops something bad from happening, so it gets said without hedging: the Ollama API has no password, no token, no user model and no permission system. Whatever can open a TCP connection to port 11434 can generate text on your CPU, enumerate the models you have pulled, pull new ones and delete the ones you use. There is no setting to turn on, because there is no mechanism to turn on.

Port 11434 is scanned continuously, for the obvious reason: an open model runner is free compute belonging to somebody else. The symptom is not an alert. It is a machine that feels slow for a fortnight and a bandwidth graph that does not match anything you did. Keep the service on loopback, and put something in front of it that asks who is calling.

If nothing outside this machine needs the model, stop here — you are already finished. Loopback plus a firewall is a complete answer, and an agent, an automation stack or a web app running on the same VPS reaches the model over 127.0.0.1 without any of what follows. Only continue if something on another machine has to call it.

The proxy, and the token. Generate a long random token, point an A record at the box, and let Caddy handle both the certificate and the door:

openssl rand -hex 32      # → the bearer token, store it in a password manager
apt install -y caddy

/etc/caddy/Caddyfile

llm.example.com {
    @unauthorised not header Authorization "Bearer PASTE_THE_HEX_TOKEN_HERE"
    respond @unauthorised "unauthorised" 401

    reverse_proxy 127.0.0.1:11434 {
        # Generation is slow by nature — do not let the proxy give up first.
        transport http {
            read_timeout 10m
        }
    }
}

systemctl reload caddy
ufw allow 80,443/tcp
ufw status                # 11434 must appear nowhere

There is a small elegance in choosing a bearer token rather than basic authentication. Ollama exposes an OpenAI-compatible API, every OpenAI client sends its key as exactly that header, and so the token you just generated becomes the API key your applications already know how to carry. Nothing needs a custom header, and rotating access is one line in the Caddyfile and one environment variable at the caller.

Verify from a machine that is not the server. The first call should be refused and the second should answer:

curl -s -o /dev/null -w '%{http_code}\n' https://llm.example.com/api/tags
# → 401

curl -s https://llm.example.com/api/tags -H "Authorization: Bearer THE_TOKEN" | jq '.models[].name'
# → the list of models you pulled

If the model is going to be called by agents rather than by you, the exposure question deserves the fuller treatment in the guide on hosting a remote MCP server — same reasoning about TLS, tokens and what should be reachable from where, applied to a service that hands tools rather than tokens.

Cap the context. The context length is the main dial on memory use, and the default is more generous than a small box wants. Set it globally to what your longest realistic prompt actually needs — most classification and extraction work is comfortable inside four thousand tokens — and raise it per request on the rare call that needs more:

Environment="OLLAMA_CONTEXT_LENGTH=4096"

curl -s http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "…a long document…",
  "options": { "num_ctx": 16384 },
  "stream": false
}'

Decide how long the model stays in memory. A resident model is instant to call and holds several gigabytes hostage. On a machine dedicated to inference, keep it loaded forever; on a shared one, let it fall out after a while. The value travels per request too, so a nightly batch can pin the model for its run and release it at the end:

OLLAMA_KEEP_ALIVE=-1     # resident until the service restarts
OLLAMA_KEEP_ALIVE=30m    # a sensible default on a shared box
OLLAMA_KEEP_ALIVE=0      # unload immediately after each request

ollama ps                # what is resident right now, and how much it holds

Serialise the queue. Two concurrent requests against one model do not go twice as fast; they contend for the same saturated memory path and each one gets slower, and if Ollama decides to load a second copy to serve them the box runs out of RAM. On this class of hardware one request at a time, with a queue in front, is both the fastest and the only safe configuration — which is why OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS were pinned to one back in step 02.

Watch the disk. Comparing four models leaves four models behind, and at five gigabytes each that is a disk filling quietly while nothing complains. Make the cleanup part of the comparison rather than a task for later:

ollama list
du -sh /var/lib/ollama/models
ollama rm mistral:7b qwen2.5:14b

Finally, back up what is not reproducible. The model weights are not — a pull brings them back. What is worth a repository is the configuration, the Caddyfile, the token, the prompts you refined over three weeks and, if you built one, the vector index behind your retrieval. The encrypted backups guide covers the mechanics; the include list for this box is short and worth writing down.

People reason about model privacy as though the risk were in the output. It is not. The output is generic text; a thousand other people got something similar. The revealing artefact is the input — and a year of inputs is a remarkably complete portrait of an organisation.

Consider what a request log actually contains. The contract you pasted in to have summarised. The customer email you asked to be classified, with the customer in it. The medical letter, the salary figure, the resignation you were drafting, the code from a repository that is not public. Then the metadata around it: which questions, in what order, at what hour, from which address, across how many months. Nobody signs a document saying "here is our strategy" — but the sequence of questions is that document, assembled honestly, by you, one call at a time.

Layer one — what the endpoint keeps. Serious providers publish serious policies, and the good ones genuinely do not train on business API traffic. That is not the same promise as not retaining it. Requests are typically stored for some period for abuse monitoring, are reachable by staff under defined conditions, and are producible under legal process — which is exactly the right way to run a large platform and exactly the wrong place for text you would not email to a stranger. A model on your own machine has no such log unless you write one.

Layer two — the identity the log is joined to. Retention alone is only half the exposure. The other half is the key it joins to: an account, a company name, a billing address, a card. That join is what turns "someone asked about acquiring a competitor" into "this company asked, on that Tuesday". Removing the model from the equation removes the retention; removing the identity from the machine removes the join. A VPS opened without an email address or an identity document, in a Nordic jurisdiction, settled in Monero, is the second half of the same move.

Layer three — the logs you write yourself. Self-hosting moves the risk rather than deleting it, and it is worth being clear-eyed about where it lands. Your application probably logs its own prompts. Your reverse proxy logs every request line. A debugging session two months ago left verbose output in the journal. The model itself keeps nothing between calls, but the machinery around it can keep everything — so decide deliberately what is written down, set retention on it, and hold that log to the standard you were unwilling to accept from someone else.

None of which makes a local model a privacy guarantee on its own. It makes it the one architecture where the guarantee is yours to give: the text stays on hardware you rent, under a jurisdiction you chose, behind a token you issued, and it is answerable to nothing else.