Running 8× H100 on-prem for clinical AI: OpenShift AI, vLLM, and PHI that never leaves the cluster
Most “how I use GPUs” posts are about a laptop with one card, or a cloud bill. This one is about a box that sits inside a hospital’s firewall: eight NVIDIA H100s on an on-premises OpenShift AI cluster, used to run open large language models against clinical data that is legally not allowed to leave the building. The hardware is the easy part. The interesting engineering is the governance — and it turns out the governance is what makes the hardware worth having.
Why on-prem at all
The reason is one sentence: clinical notes cannot go to a cloud API. The moment your workload needs free-text notes — discharge summaries, radiology impressions, nursing narratives — a hosted endpoint is off the table for a covered entity. You either don’t do NLP on notes, or you bring the model to the data. On-prem frontier-capable GPUs are what let the second option exist. This box isn’t nostalgia for owning metal; it’s the only configuration in which you can run a 70B open model over protected notes at all.
The hardware, briefly
Eight H100s is 640 GB of VRAM, FP8-capable. That number is what matters, not the FLOPS:
- A 70B model in FP8 fits comfortably with room for a large KV cache, so you get real batched throughput, not a single-stream toy.
- Inference is bandwidth-bound, not compute-bound — the H100’s HBM bandwidth is why tokens come out fast; the tensor cores are rarely the bottleneck at inference.
- 640 GB total means you can hold more than one model resident, or one big model with long context and high concurrency.
FP8 is the unlock: it roughly halves the memory footprint versus FP16 with negligible quality loss for serving, which is the difference between “70B barely fits” and “70B fits with headroom for throughput.”
Three API layers (the mental model)
The single most useful thing for staying oriented on OpenShift AI is to stop thinking of it as “Kubernetes” and instead as three stacked APIs, each with a different audience:
| Layer | API | Who uses it |
|---|---|---|
| 1 | Cluster API | pods, PVCs, routes, builds — the platform plumbing |
| 2 | AI Operator API | notebooks, pipelines, model registry — the ML control plane |
| 3 | Model Serving / Inference API | OpenAI-compatible LLM endpoints |
The point of the split: downstream consumers only ever touch Layer 3. A cohort-builder tool or a Shiny app doesn’t know or care that there’s a Kubernetes cluster underneath — it sees an OpenAI-compatible URL and a token. All the platform complexity lives in Layers 1–2 and is the operator’s problem, not the analyst’s. Designing so consumers depend only on Layer 3 is what keeps the whole thing maintainable.
Serving: vLLM, OpenAI-compatible
Serving is vLLM, provisioned through the OpenShift AI operator, exposing an OpenAI-compatible API. That compatibility is the quiet superpower: every client library, every base_url swap, every tool that already speaks the OpenAI protocol works unchanged. You point existing code at the on-prem endpoint and it just runs — the model is a Gemma or a Llama-family open model instead of a hosted one, and nothing downstream has to know.
The governance design that actually matters
Here is the part worth copying. PHI authorization is not one switch; it’s three locations with three different rules, and the whole architecture falls out of respecting them:
- The cluster (inside the perimeter) is authorized for PHI including clinical notes. This is the reason the box exists — it’s the one place notes are allowed.
- The jump box (inside the firewall) may hold structured PHI (warehouse extracts, tabular EHR) but not clinical notes.
- The developer workstation (outside the firewall, where the coding agent runs) gets no PHI at all.
Everything else is a consequence:
- Notes flow warehouse → cluster Pod → warehouse, and never anywhere else. Nothing notes-derived comes back through
oc logs, and nothing rides out on a git round-trip. - The workstation authors code, not data. It holds manifests, scripts, and connection config — never a row of PHI. The model comes to the data; the developer never sees the data.
- Structured PHI and free-text PHI are treated as different classifications with different blast radii, because they are.
If you take one idea from this post, take that one: write the data-residency boundaries down as locations-with-authorizations first, and let the deployment topology be derived from them — not the other way around. It makes every later “can this go here?” question a lookup instead of a debate.
Driving it from a laptop outside the firewall
Day-to-day, the cluster is driven from a workstation that is not allowed to hold PHI — which is exactly why the workflow is all control-plane, no data-plane:
- A Nix dev shell pins
oc,kubectl,helm,jq,yqso the toolchain is reproducible and nothing is installed globally. - Auth is a short-lived console token in a gitignored
.env— expect to refresh it whenever a login check fails. - A read-only connectivity test verifies reachability and RBAC across resource categories without creating anything: it checks the API is reachable, the token logs in, and read access holds on pods/services/PVCs/routes — and it reports the expected denials (e.g.
secretsrestricted, serving CRDs not yet granted) as SKIPs rather than failures.
That last habit is underrated: a connectivity script that knows which denials are expected turns “is it broken or is it RBAC?” from a 30-minute rabbit hole into a line of output. On a locked-down namespace you spend more time reasoning about permissions than about pods, so make the permissions legible.
What it’s actually for
The hardware is a means. The workloads are the point:
- Clinical NLP → structured concepts — turning free-text notes into coded observations, the canonical task that requires notes to stay on the cluster.
- Text-to-SQL grounded on a real schema catalog — a model that writes warehouse queries, grounded on an exported catalog of the actual tables so it doesn’t hallucinate columns.
Both are things you cannot buy as a hosted API without either shipping PHI off-site or accepting a model that has never seen your schema. On-prem H100s + open models + a catalog is how you get frontier-ish capability and keep the data where the law says it has to stay.
The takeaway
Owning eight H100s isn’t the flex. The flex is that the governance is designed before the deployment, so the expensive hardware is pointed at the one class of problem — protected clinical text — that no cloud endpoint can legally touch. If you’re standing up on-prem AI for healthcare: buy the VRAM, yes, but spend your real design time on the three-location residency model and the Layer-3-only consumer contract. That’s what turns a GPU box into a platform.