This white paper sketches a reference architecture for running a production large-language-model assistant entirely inside your own perimeter, including the fully air-gapped case. It is deliberately vendor-neutral in its principles, with notes on how Kaldryn implements each piece.
Design goals
- Zero data egress, no inference input, output or document ever leaves the perimeter.
- No phone-home, licensing, updates and operation work without outbound internet.
- Single-server start, cluster-ready growth, scale out without rebuilding.
- Defence in depth, isolation enforced at multiple independent layers.
- Operable by a small team, no dedicated platform or DevOps org required.
The five planes
It is useful to decompose the appliance into five planes, each with a clear responsibility:
- Serving plane, the model runtime (e.g. vLLM and Ollama behind a load balancer) with tier-aware routing across CPU and GPU nodes.
- Retrieval plane, document ingestion, embeddings, a vector store and security-trimmed RAG so answers cite your own files.
- Identity plane, SSO/SAML, group-to-role mapping and just-in-time provisioning, with metadata upload for air-gapped IdPs.
- Governance plane, DLP scanning, an immutable audit log, and policy enforcement on every inbound and outbound message.
- Observability plane, latency percentiles, tokens/sec, queue depth and KV-cache utilisation per tier.
Keeping these planes distinct matters operationally: you can scale the serving plane (add GPUs) independently of the retrieval plane (add storage and embedding throughput) as usage grows.
Network topology
For the air-gapped deployment, the topology is intentionally boring. Clients reach the appliance over the internal network only; the appliance has no default route to the internet. Where an organisation wants connected convenience for some workloads and isolation for others, the recommended pattern is a single appliance running in data-isolation mode with egress sealed, rather than two separate systems to keep in sync.
Air-gapping should never rely on one toggle. We recommend three independent layers:
- Application layer, outbound networking disabled at the code level so the software cannot initiate external calls.
- Host layer, an egress firewall (e.g. nftables) that drops outbound traffic regardless of application behaviour.
- Network layer, no route to the internet from the appliance's VLAN, verified by the customer's own controls.
GPU sizing
Sizing is driven by three variables: the parameter count and quantisation of the models you intend to serve, your target concurrency, and your latency budget (time-to-first-token and tokens/sec). As rough guidance, always validate with your own benchmark on your own hardware:
| Deployment | Typical hardware | Good for |
|---|---|---|
| Pilot / small team | 1 server, 1-2 modern data-centre GPUs | Up to a few dozen active users, mid-size models |
| Department | 1-2 servers, 4-8 GPUs | Hundreds of users, larger models, RAG at scale |
| Organisation | Multi-node GPU cluster, load-balanced | 500+ users, tiered model routing, high concurrency |
The important architectural property is that moving between these tiers is a capacity change, not a re-platforming. Tier-aware routing lets you add nodes and point heavier workloads at GPU tiers while lighter ones stay on cheaper hardware.
Model serving and provenance
Each model should be verified by content hash on first load and pinned thereafter, so the weights you reviewed are the weights that run. A provenance grade, covering license, source, stability and integrity, lets administrators make informed choices from a curated catalogue rather than pulling arbitrary weights from the internet. We cover this in depth in our note on model provenance.
Governance on the hot path
DLP and audit are not bolt-ons; they sit on the message hot path. A regex-and-rules scan runs on every inbound and outbound message to block, redact or audit PII, secrets and network data, and every action is written to an immutable, SIEM-exportable log. Because nothing leaves the perimeter, this audit trail is also the organisation's complete record for eDiscovery and supervisory review.
Deployment and day-two operations
The reference deployment is a single signed, checksummed installer that carries every image, model and dependency, runs preflight checks, stands up services and warms a default model, no manual wiring. For air-gapped sites, updates arrive as signed media rather than over the network, and restores run detached so the appliance cannot be bricked by a bad upgrade. The goal throughout is that a small IT team, not a platform org, can own it.
Want the full version of this reference architecture, including diagrams and a sizing worksheet? Talk to our team, we provide it under NDA for evaluation.