This runbook walks through installing the Nebari LLM serving pack onto a
fresh Nebari cluster managed by ArgoCD. The validation that produced this
runbook ran on AWS (NIC + EKS); the steps are written so they should
work on any cloud where NIC supports a foundational deploy, but only AWS
has been exercised end-to-end so far. For local development against
kind see Local Development instead.
What “fresh” means here: every command in this document was run, in
order, against a brand-new Nebari Infrastructure Core (NIC) deployment
with no hand-applied patches, no leftover state, and no cluster-side
workarounds. If you need a manual step that is not in this document,
that is a bug in this document - please open an issue.
This runbook starts from a NIC-foundational Nebari cluster. NIC ships,
out of the box, the components below; if any are missing or in a
different namespace on your cluster, adjust the commands accordingly.
Component
Namespace
Purpose
ArgoCD
argocd
GitOps controller. New apps in this runbook get installed via ArgoCD Application manifests committed to your cluster-config repo.
cert-manager
cert-manager
Issues TLS certs for the gateway and for pack-managed Certificates.
Envoy Gateway
envoy-gateway-system
Gateway API data plane. Will be reconfigured in section 5 to wire up the AI Gateway extension manager.
Keycloak
keycloak
OIDC provider. Provides the groups claim to JWT-protected endpoints. The admin Secret is keycloak-admin-credentials.
Longhorn
longhorn-system
Default StorageClass longhorn (RWX). Used for the model PVC.
AWS Load Balancer Controller
kube-system
Provisions NLBs for the Gateway resources (AWS only).
Nebari Operator
nebari-operator-system
Provisions NebariApps. NIC pre-wires its KEYCLOAK_* and TLS_CLUSTER_ISSUER_NAME env vars, so you do not need to set them by hand.
Nebari landing page
nebari-system
The tile-based home page; the pack’s key-manager UI is exposed as a NebariApp tile.
OpenTelemetry Collector
monitoring
Telemetry sink. Optional.
In addition you need:
At least one GPU node in the cluster. AWS users: a g6e.xlarge
(1x L40S, 48 GB VRAM) or larger node from a node group running the
AL2023 NVIDIA AMI is the validated baseline. The NVIDIA device
plugin will be installed in section 3, so the node will not yet
show nvidia.com/gpu in its capacity.
NVIDIA driver 580 or later on every GPU node. As of llm-d
v0.7.0 the serving images (llm-d-cuda:v0.7.0) ship the CUDA 13.0.2
runtime, which requires driver branch 580+. Nodes on an older driver
must be upgraded before deploying this pack, or the vLLM container
will fail to start with a CUDA driver/runtime version mismatch.
Confirm with nvidia-smi on the node (or
kubectl exec into the GPU operator’s driver/validator pod): the
“Driver Version” must be >= 580.
DNS zone control over <baseDomain> with a wildcard CNAME
pointing every *.<baseDomain> at the Gateway’s load balancer.
HTTP-01 ACME challenges will run against llm.<baseDomain> and
llm-internal.<baseDomain>, so both must resolve before you start.
A Cluster Issuer that cert-manager can use for HTTP-01. NIC
ships one; the validated install used letsencrypt-issuer
(the chart’s default letsencrypt-production is a different name -
you will set platform.tls.clusterIssuer to match in section 8).
A cluster-config git repo that ArgoCD’s AppProject is configured
to read from. New Application manifests in this runbook will be
committed there; the path is up to you (e.g. clusters/<name>/apps/).
Note on existing workarounds: if you are coming from the v0
install path on an older cluster, the runbook deliberately does not
document the keyManager.image.tag Argo override or the manual NLB
security-group ingress rules. Those were workarounds for the
llmd-test1 reference cluster and do not apply on a fresh
NIC-foundational deploy. If you find yourself needing one, that is
a regression - open an issue.
Expected: a context name that matches your fresh cluster, and at least
one node whose instance-type is in the g5/g6/g6e family (or your cloud’s
GPU equivalent). If the GPU node does not appear, your node group is
not provisioned yet - fix that before continuing.
2.2 Confirm ArgoCD is healthy and the AppProject exists
Expected: every ArgoCD pod is Ready; an AppProject named foundational
(or whichever project NIC put new apps into) is present. Note its name -
you will set it as spec.project on every new Application below.
Expected: every hostname resolves to $GATEWAY_LB (or the LB’s IP). If
any do not, check your wildcard CNAME and DNS propagation; do not
proceed - HTTP-01 challenges will fail and the install will stall.
Expected: every Certificate is READY=True, and the curl returns an
HTTP status (any 4xx is fine - it just means no route matches yet, but
TLS handshake worked). If you see “connection reset by peer”, the
gateway has no usable cert. The known fresh-install case where this
happens is documented in section 12.1 (NIC nebari-gateway-cert stuck
on first install).
2.6 Confirm Keycloak is reachable internally and externally
Expected: the service exists in the keycloak namespace; the external
URL returns HTTP 200 with a JSON body. If the external URL fails, the
Gateway/cert chain is not yet wired up - go back to step 2.5.
2.7 Confirm the nebari-operator has the expected Keycloak env vars
If any are missing, NIC’s nebari-operator deployment is mis-configured.
Fix that on the NIC side before continuing - the pack relies on the
operator being able to mint Keycloak clients.
The pack’s model pods request nvidia.com/gpu from Kubernetes. NIC’s
GPU node group runs the AL2023 NVIDIA AMI which already ships the
kernel driver, so all that is needed is the container toolkit and the
device plugin. The NVIDIA GPU Operator chart installs both, plus
node-feature-discovery so GPU nodes get the right labels.
Driver version (llm-d v0.7.0): the AMI’s pre-installed driver must
be branch 580 or later, because the llm-d-cuda:v0.7.0 serving images
use the CUDA 13.0.2 runtime. If your AMI ships an older driver, either
use a newer AMI or let the GPU Operator manage the driver
(driver.enabled=true) pinned to a 580+ branch. Verify after the
operator is up with kubectl logs -n nvidia-gpu-operator <driver-or-validator-pod>
or nvidia-smi on the node.
Skip this subsection on a managed node group that ships a vendor GPU
AMI (e.g. EKS + AL2023 NVIDIA AMI). It applies when the cluster is
k3s on hosts you manage yourself (on-prem, bare-metal, or a k3s
test rig). Validated on Ubuntu 24.04 with an A10G; the same applies to
the on-prem RTX A5000 (both are GA102, same -server-open driver
branch).
On k3s two things differ from the managed-AMI path, and both must be
handled before continuing to 3.1:
1. Disable the operator’s toolkit as well as its driver, and install
both on the host. The operator’s nvidia-container-toolkit-daemonset
assumes a vanilla-containerd layout and overwrites k3s’s generated
config.toml (replacing the full CRI config with a stub), which knocks
the node NotReady. Setting toolkit.env CONTAINERD_CONFIG=... does
not help - it triggers the same clobber. Use this values block in
the section 3.1 Application instead of the driver.enabled: false-only
one:
values: |
# Driver and toolkit are installed on the host (apt). The operator
# only runs NFD + device-plugin + dcgm to expose nvidia.com/gpu.
If a previous operator-managed toolkit install left a binary behind,
remove it (sudo rm -rf /usr/local/nvidia/toolkit) so k3s points at
/usr/bin/nvidia-container-runtime on restart.
2. Make nvidia the default containerd runtime. The pack’s model
pods are created with no runtimeClassName, so on k3s (whose
default runtime is runc) the GPU libraries are never injected and vLLM
fails with Failed to infer device type. k3s auto-detects a
host-installed nvidia-container-runtime and writes an nvidia
RuntimeClass into its generated config, but does not make it the
default. Pin it with a single self-contained/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl, then
sudo systemctl restart k3s:
version = 2
[plugins."io.containerd.internal.v1.opt"]
path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
stream_server_address = "127.0.0.1"
stream_server_port = "10010"
enable_selinux = false
enable_unprivileged_ports = true
enable_unprivileged_icmp = true
sandbox_image = "rancher/mirrored-pause:3.6"
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "overlayfs"
disable_snapshot_annotations = true
default_runtime_name = "nvidia"
[plugins."io.containerd.grpc.v1.cri".cni]
# NOTE: this data-dir hash is k3s-version-specific. Copy the bin_dir
# value from the live /var/lib/rancher/k3s/agent/etc/containerd/config.toml
Do not try to set only default_runtime_name by re-opening the
[plugins."io.containerd.grpc.v1.cri".containerd] table on top of
k3s’s {{ template "base" . }} include. Reopening an existing table
produces a duplicated tables TOML error that crashes containerd and
takes the node NotReady. Appending a brand-new sub-table is legal;
reopening an existing one is not. Use a single static template like
the one above. Validate it with
python3 -c "import tomllib; tomllib.load(open('config.toml.tmpl','rb'))"before restarting k3s.
# AL2023 NVIDIA AMI ships the kernel driver; only toolkit +
# device plugin are needed.
driver:
enabled: false
destination:
server: https://kubernetes.default.svc
namespace: nvidia-gpu-operator
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
git push the file. ArgoCD’s nebari-root app-of-apps picks it up on
its next refresh (typically within a minute; you can force it with
kubectl annotate application -n argocd nebari-root argocd.argoproj.io/refresh=hard --overwrite).
Why this exact chart version: versions before v25.10.x of the
gpu-operator chart render spec.validator.plugin: null in the
auto-generated ClusterPolicy/cluster-policy, which the operator’s
own admission webhook rejects with Invalid value: "null": spec.validator.plugin in body must be of type object. v25.10.1 is
the lowest version validated against this runbook.
The ArgoCD Application may report OutOfSync even when everything is
working. This is a known artifact of the gpu-operator chart’s
PreSync/PostSync hooks: hook objects are created and pruned during
each sync, leaving ArgoCD’s last-observed manifest set out of sync
with the live state. The operative signal is the pod set above.
Expected: kubectl logs gpu-smoke prints something like
GPU 0: NVIDIA L40S (UUID: GPU-...). If the pod sits Pending, the
device plugin is not reporting GPUs to the kubelet; check
kubectl describe node <gpu-node> for nvidia.com/gpu under
Capacity and Allocatable, and kubectl logs -n nvidia-gpu-operator <device-plugin-pod> for errors.
k3s / host-managed runtime (section 3.0): if the pod runs but
the logs show exec: "nvidia-smi": executable file not found in $PATH (or Failed to infer device type), the pod was scheduled on
the default runc runtime, which does not inject the NVIDIA
libraries/binaries. Either make nvidia the default runtime (as in
the section 3.0 config.toml.tmpl), or add runtimeClassName: nvidia to the pod spec (spec.runtimeClassName: nvidia, a sibling
of restartPolicy). The GPU Operator’s device plugin reports
nvidia.com/gpu regardless of which runtime is default, so a pod
can schedule onto the GPU yet still miss the NVIDIA injection.
4. Install AI Gateway and Inference Extension CRDs
The pack’s per-model routing relies on two upstream CRD bundles that
must exist on the cluster before the pack itself reconciles any
LLMModel:
Envoy AI Gateway CRDs (AIGatewayRoute, AIServiceBackend,
BackendSecurityPolicy, GatewayConfig, MCPRoute) - the
nebari-llm-operator creates these per LLMModel and the AI
Gateway controller (section 6) reconciles them.
the llm-d End-Point Picker (EPP) container looks up
InferencePool at startup and crashloops without it.
Both bundles install the CRDs only; controllers come in section 6 (AI
Gateway) and are not required for the inference-extension on this
runbook (the EPP is bundled inside the LLMModel pod the operator
creates).
clusters/<name>/apps/gateway-api-inference-extension.yaml - this one
points at a kustomize directory in your repo (because the upstream
release ships a flat manifests.yaml that ArgoCD cannot Helm-template):
The AI Gateway controller does two jobs the pack relies on:
XDS extension server. Envoy Gateway calls it during XDS
translation (over a gRPC service on port 1063) to insert the
ext_proc HTTP filter into the listener filter chain for routes
that reference an InferencePool or AIGatewayRoute. Without this,
per-model routing falls back to direct_response: 500.
Pod-mutating admission webhook. When Envoy Gateway creates the
Envoy proxy pod (the data plane), the webhook patches in an
ai-gateway-extproc native-sidecar container. The webhook only
injects when there is at least one AIGatewayRoute bound to the
gateway, so the sidecar will appear on the next proxy-pod recreation
after the first LLMModel reconciles in section 9.
Install the controller before the envoy-gateway reconfig in section 6:
envoy-gateway’s extensionManager.service.fqdn will point at this
controller’s Service, and bringing them up in this order avoids noisy
“connection refused” log lines during XDS translation.
Port 1063 is the XDS extension server (envoy-gateway’s
extensionManager.service.fqdn will reference this). Port 9443 is the
admission webhook. 9090 is metrics.
6. Reconfigure envoy-gateway with AI Gateway extension wiring
Envoy Gateway needs three configuration additions that NIC’s default
install does not include:
extensionApis.enableBackend: true so it accepts
inference.networking.k8s.io/InferencePool as a valid HTTPRoute
backend kind.
A full extensionManager block that points at the AI Gateway
controller’s XDS extension server (port 1063) and asks Envoy
Gateway to call into it on every listener / route / cluster /
secret translation step.
backendResources enumerating which non-builtin backend kinds the
extension manager handles, so Envoy Gateway does not reject routes
that reference them.
Locate the file in your cluster-config repo where NIC’s
envoy-gateway ArgoCD Application lives. NIC ships it with a minimal
spec.source.helm.values block: just controllerName, deployment
resources, and Service type. Merge in the new keys without removing
any existing ones:
# ... preserve any existing deployment / service / podDisruptionBudget
# values that NIC was already setting; only the config.envoyGateway
# subtree changes.
git push. ArgoCD will sync the chart with the new values; the
envoy-gateway-config ConfigMap is updated, but the running
controller process does not pick up changes until the deployment
restarts.
Expected: rollout completes; new envoy-gateway pod becomes Ready
within ~30 seconds. Existing HTTPRoutes (for argocd, keycloak, the
landing page) keep serving from the existing Envoy proxy pod
throughout - the controller restart only blocks new XDS pushes.
Expected: the rendered config shows the extensionManager block with
hostname: ai-gateway-controller.envoy-ai-gateway-system...,
port: 1063, and backendResources listing InferencePool.
The rendered envoy-gateway.yaml sorts top-level keys
alphabetically, so hostname/port land ~20 lines below the
extensionManager: header. grep -A 5 truncates the block before
them - use -A 25 and anchor on ^extensionManager: so you match
the block header, not the inline backendResources reference.
Note on the Envoy proxy pod. You do not need to recreate the
existing Envoy proxy pod (the data plane) at this point. The AI
Gateway pod-mutator only injects the extproc sidecar when at least
one AIGatewayRoute is bound to the gateway. The sidecar will
appear automatically the next time Envoy Gateway recreates the
proxy pod after the first LLMModel reconciles in section 9. If
you want to verify the injection sooner, you can apply a placeholder
AIGatewayRoute and delete the proxy pod, but it is not required.
Beta documentation gate: This section covers the auth setup that the pack requires before it will reconcile any LLMModel. Both the Keycloak group (7.2) and the operator environment check (7.1) must pass before proceeding to section 8.
The pack expects two things on the Keycloak side:
A group whose name matches whatever you put in
LLMModel.spec.access.groups. This runbook uses llm.
The nebari-operator deployment carrying the right KEYCLOAK_*
environment so it can mint a Keycloak client for the key-manager
NebariApp. NIC pre-wires this on a foundational deploy; we just
verify.
The Keycloak admin Secret on a NIC deploy is keycloak-admin-credentials
in the keycloak namespace. Note its key names: admin-username and
admin-password (NOT username/password). Fetch a token via the
admin-cli client, then call the realm admin API:
Expected: a JSON array with one entry having "name": "llm" and a
non-empty id.
Why this is a manual step. The pack does not create groups; it
only references them. Letting the operator manage groups would
couple it tightly to a single IdP’s API surface, working against the
(still-pending) provider-agnostic OIDC discovery (#66). Group
creation belongs in your IdP-of-record’s source of truth, whether
that is the Keycloak admin UI, an IaC layer, or this curl.
The pack itself ships as a single Helm chart that reconciles three things
into the cluster:
The pack operator (nebari-llm-serving-operator) which watches
LLMModel CRs and renders the corresponding Deployment, Service,
HTTPRoutes, AIGatewayRoute, AIServiceBackend, InferencePool +
InferenceModel, and (with manageSharedListeners: true) two
Gateway listeners + a shared TLS Certificate.
The key-manager (nebari-llm-serving-key-manager) which is the
user-facing UI for minting API keys, plus the back-end that
validates keys on inbound traffic to the external gateway.
A NebariApp for the key-manager which the NIC nebari-operator
reconciles into a Keycloak OIDC client + an HTTPRoute on the public
Gateway + a tile on the landing page.
This is sync-wave 7: after cert-manager, envoy-gateway, AI Gateway
controller, inference-extension CRDs, GPU operator, Keycloak, the NIC
nebari-operator, and the landing page have all converged.
The platform.gateway.external and platform.gateway.internal blocks
both point at nebari-gateway because this runbook uses a single
shared Gateway. With manageSharedListeners: true the operator adds
two new listeners (llm-https for external, llm-internal-https for
internal) onto NIC’s existing nebari-gateway, each pinned to a
different hostname (llm.<baseDomain> and
llm-internal.<baseDomain>) and a different SecurityPolicy
(API-key-protected on external, JWT-protected on internal).
If you want to split traffic across two physical Gateways instead
(e.g. one with a public LB, one with an internal-only LB), point
platform.gateway.internal at a different Gateway resource.
The key-manager NebariApp should be reconciled by the NIC
nebari-operator, which means there is also an HTTPRoute for the
key-manager UI on the public Gateway plus a Keycloak client and a
landing-page tile:
The first three listeners come from NIC; the bottom two
(llm-https, llm-internal-https) are added by the pack operator.
A shared TLS Certificate covering both hostnames lands in the pack
namespace:
The key-manager UI is reachable at https://llm-keys.<baseDomain>/,
gated by Keycloak OAuth2 (members of the llm group only). Hitting
that URL in a browser at this point should redirect through the
Keycloak login screen and bounce back to a (mostly empty) key-manager
page. There are no LLMModels yet; section 9 changes that.
An LLMModel is the user-facing API of the pack: one CR per model
you want served. Applying it triggers the pack operator to reconcile
the full per-model serving stack:
A Deployment running vllm against the model weights, with a
model-downloader init container that pulls the weights from
Hugging Face into a PVC at /model-cache on first start.
A Service fronting the vLLM container on port 8000.
A second Deployment + Service for the endpoint-picker pod
(EPP) that the gateway-api-inference-extension uses to route
requests across replicas.
An InferencePool referencing the EPP, plus matching labels so
the inference extension can find the vLLM pods.
Two HTTPRoute + AIGatewayRoute pairs (external + internal),
each pinned to a different listener on the shared Gateway.
Two SecurityPolicy resources, one per route: API-key auth on the
external route, JWT auth on the internal route.
The pack ships example LLMModel manifests under examples/models/ in
the pack repo. For this runbook the example is
Qwen/Qwen3.5-35B-A3B-GPTQ-Int4: a 35B-param mixture-of-experts model
quantized to 4-bit GPTQ that fits comfortably on a single L40S
(48 GB VRAM) with ~17.5 GB for weights and the rest for KV cache.
Pick a different model if your hardware demands it; sizing rules of
thumb:
24 GB GPUs (A10G / A5000):Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 does
NOT fit. It is a multimodal MoE - vLLM builds a dummy vision encoder
in qwen3_vl.py during profile_run, and the int4 weights plus that
encoder consume ~21.7 GiB before any KV cache, so it OOMs
(torch.OutOfMemoryError: CUDA out of memory) on a 24 GB card. For
24 GB GPUs use a small text-only model to validate the
deploy/shard/serve path, e.g. Qwen/Qwen2.5-1.5B-Instruct (fp16, no
quantization): ~2.9 GiB weights, ~16 GiB KV cache, serves cleanly at
--max-model-len 8192. See the sizing-validation note in section 9.2.
Total model weights size + ~30% headroom must fit in GPU VRAM.
For PVC-backed storage, set spec.model.storage.size to at least
twice the on-disk weights size (Hugging Face writes incomplete
shards alongside finished ones during download).
For pvc-backed huggingface downloads on a small instance type, the
model-downloader streams weights directly into the PVC, so host
RAM is not the bottleneck. The vLLM container later reads from
/model-cache via memory-mapped I/O.
metadata.namespace MUST be nebari-llm-serving-system (the pack
operator only watches its own namespace; see #59). access.groups
must list a Keycloak group that exists in the realm. endpoints. external.subdomain becomes the public hostname:
<subdomain>.<baseDomain> is what end users hit on the external
route. Internal endpoints share a single llm-internal.<baseDomain>
hostname and route by URL path under /v1/.
Apply directly with kubectl apply -f. Per-model manifests are not
gated by sync waves and do not need to live in the cluster-config
repo; treat them as data-plane content the pack consumes.
k3s / host-managed driver (section 3.0), llm-d-cuda image: add
the two env vars below under spec.advanced.vllm.extraEnv, or the
vLLM pod crashloops with ld: cannot find -l:libcuda.so.1 -
surfacing misleadingly as Model architectures [...] failed to be inspected because Triton’s JIT import crashes during vLLM’s
model-arch inspection.
Why: with the host-managed runtime the pod gets only the utility
driver capability (enough for nvidia-smi, not libcuda.so.1);
NVIDIA_DRIVER_CAPABILITIES=all injects libcuda.so.1 - but it
lands in /usr/lib/x86_64-linux-gnu (Debian multiarch) while the
image’s Triton links it RHEL-style with -l:libcuda.so.1 -L/usr/lib64, and ld searches -L dirs, not the ldconfig cache.
LIBRARY_PATH makes gcc/ld add the multiarch dir. extraEnv on the
CR is the only durable place - a kubectl set env on the Deployment
is reverted by the operator within seconds.
You’re done when the vllm container logs Started server process and
Application startup complete, the pod goes 1/1 Ready, and the
LLMModel reports Phase: Ready with Replicas: 1.
Expected: a JSON response listing the served model with the same
name from the LLMModel spec. End users hit the external and
internal routes through the gateway, with auth enforced; section 11
walks through both user journeys.
Run this checklist top-to-bottom once sections 1-9 have all
converged. Each item is independent and can be run on its own. The
checklist verifies the install at the cluster level; section 11
verifies it at the end-user level.
Expected: every Application in the foundational + pack set is
Synced/Healthy. Anything OutOfSync or Degraded blocks the rest
of the checklist; resolve before continuing.
NIC must ship with cert-manager.maxConcurrentChallenges: 1 to keep
HTTP-01 challenges from racing on shared SANs (see
nebari-dev/nebari-infrastructure-core#267 / #259). Older NIC
versions hit a race that leaves nebari-gateway-cert stuck on
failedIssuanceAttempts; if that happens, deleting the Certificate
lets ArgoCD’s selfHeal recreate it cleanly.
Expected: each envoy-proxy pod lists at least envoy,ai-gateway-extproc
among its container names. If ai-gateway-extproc is missing, force a
proxy-pod restart (kubectl rollout restart deploy -n envoy-gateway-system <proxy-deploy>)
to give the webhook another shot.
Expected: a JSON completion with non-empty choices[].text. This
confirms vLLM has the weights loaded and is generating tokens; auth
and gateway routing are validated separately in section 11.
Beta documentation gate - dual-endpoint auth: The pack uses two
different Keycloak URL forms for the SecurityPolicy OIDC config.
Browser-facing endpoints (authorizationEndpoint, endSessionEndpoint)
must use the public https://keycloak.<baseDomain>/... URL so that
browser redirects resolve. Back-channel endpoints (tokenEndpoint,
issuer) use the in-cluster http://keycloak-keycloakx-http.keycloak.svc.cluster.local:8080/...
URL for performance and to avoid hairpinning through the public Gateway.
This split is implemented in nebari-operator >= v0.1.0-alpha.19. If
your cluster has an older operator, section 12.2 documents the
workaround.
In a fresh browser session (incognito works well to avoid stale
cookies), open:
https://<baseDomain>/ -> NIC landing page should render with a
tile for nebari-llm-serving-key-manager.
https://llm-keys.<baseDomain>/ -> Keycloak login screen, then
the key-manager UI for users in the llm group.
If the browser dead-ends on keycloak-keycloakx-http.keycloak.svc.cluster.local,
re-run check 10.9.
If every check passes, the install is good. Section 11 walks
through the actual end-user journeys (mint a key, hit the external
endpoint, and verify that non-allowed users are denied).
Expected: HTTP 200 with a JSON body containing
choices[0].message.content. The model value in the request body
must match spec.model.name from the LLMModel CR (the full
HuggingFace model ID, e.g. Qwen/Qwen3.5-35B-A3B-GPTQ-Int4), not
the LLMModel metadata name.
Log into https://llm-keys.<baseDomain>/ as outsider@example.com.
The “Available Models” section should be empty - the outsider cannot
see any models and therefore cannot mint an API key.
If the outsider somehow obtains a key or constructs a direct API
call, the key-manager API returns HTTP 403:
Terminal window
# Via port-forward (bypasses gateway auth to test the key-manager directly):
These are issues discovered during the fresh-install validation that
require manual intervention until upstream fixes ship. Each links to
a tracking issue. Once the fix ships, the pre-step can be removed
from this runbook.
Symptom: After a fresh NIC deployment, nebari-gateway-cert in
envoy-gateway-system stays Ready=False with a failed ACME order.
The gateway’s HTTPS listener has no usable cert and external
connections are refused.
NIC should ship with cert-manager.maxConcurrentChallenges: 1 to
prevent HTTP-01 solver races on overlapping SANs
(nebari-dev/nebari-infrastructure-core#259).
12.2 SecurityPolicy uses in-cluster Keycloak URLs for browser-facing endpoints
Symptom: The key-manager UI redirects the browser to
http://keycloak-keycloakx-http.keycloak.svc.cluster.local:8080/...
instead of the public https://keycloak.<baseDomain>/... URL. The
OAuth2 flow dead-ends because browsers cannot resolve in-cluster
hostnames.
Fix: Requires nebari-operator >= v0.1.0-alpha.19 (PR #111).
This version uses KEYCLOAK_EXTERNAL_URL for
authorizationEndpoint and endSessionEndpoint (the two endpoints
the browser hits) while keeping tokenEndpoint and issuer as
in-cluster URLs (back-channel only).
If your NIC ships an older nebari-operator, override the image in
your NIC kustomization or ArgoCD values until the fix is released
upstream.
12.3 AI Gateway webhook certificate becomes untrusted after pod rescheduling
External connectivity is completely down (NLB target groups have zero
registered targets).
Root cause: The AI Gateway controller generates a self-signed CA
at startup and patches the MutatingWebhookConfiguration with the CA
bundle. When the controller pod is rescheduled (e.g. after a node
replacement), the new pod may generate a new CA while the webhook
config retains the old one.
# The envoy proxy pod should recover within ~30 seconds
kubectlgetdeploy-nenvoy-gateway-system-w
Note: This needs further investigation to determine whether it
is an upstream AI Gateway bug or a configuration issue. A tracking
issue will be filed once the root cause is confirmed.
The Envoy proxy is returning a 500 before the request reaches the
AI Gateway extension processor. Common cause:
extensionApis.enableEnvoyPatchPolicy or
extensionApis.enableBackend is missing from the envoy-gateway
config. Re-check section 6.
Authentication is pooled across every model’s api-keys Secret, so a 401
means the key is not present in any model’s Secret (revoked, mistyped, or
never minted).
Verify the API key Secret exists: kubectl get secret -n nebari-llm-serving-system <model>-api-keys
Verify the model field in your request body matches the
HuggingFace model ID (e.g. Qwen/Qwen3.5-35B-A3B-GPTQ-Int4),
not the LLMModel CR name
Keys authenticate anywhere on the shared listener but are authorized only
for the model they were minted for, so a 403 means the key is valid but
not on this model’s allow-list.
Verify the key was minted for the model you are calling (a key for
model A returns 403 against model B by design)
A freshly minted key can 403 until the operator re-renders the model’s
SecurityPolicy allow-list - typically seconds, within about a minute;
retry before digging deeper
Key-manager UI shows “No models available” for a user who should have access
Check kubectl describe pod <pod> for scheduling failures
Common causes: no GPU node available, PVC not bound (wrong
storageClass), insufficient CPU/memory
For single-GPU nodes: only one model can run at a time. A rolling
update will deadlock if the new pod cannot schedule alongside the
old one (the new pod stays Pending while the old pod holds the
one GPU). On the nebari pack the operator enforces
spec.serving.replicas, so scaling the Deployment to 0 (or setting
replicas: 0) is reverted within seconds. Instead, after applying
the respec, kubectl delete pod <old-pod> to free the GPU and let
the new pod schedule.
The init container downloads the model from HuggingFace on first
deploy. Large models (30GB+) can take 10-20 minutes on typical
network connections. Check the init container logs:
The AI Gateway mutating webhook only injects the sidecar when an
AIGatewayRoute exists and the webhook is healthy. If the proxy pod
was created before the AI Gateway controller was ready, delete the
proxy pod to trigger re-injection:
Seen on k3s / host-managed driver nodes (section 3.0) running the
llm-d-cuda image. The traceback often surfaces as Model architectures [...] failed to be inspected because Triton’s JIT
import crashes during vLLM’s model-arch inspection. The pod has only
the utility driver capability, and even with libcuda.so.1
injected it lands in a directory ld does not search. Fix: add
NVIDIA_DRIVER_CAPABILITIES=all and
LIBRARY_PATH=/usr/lib/x86_64-linux-gnu to
spec.advanced.vllm.extraEnv on the LLMModel - see the callout in
section 9.2.
A related symptom on the same nodes is vLLM logging Failed to infer device type: the pod was scheduled on the default runc runtime,
which exposes no GPU. Make nvidia the default containerd runtime or
ensure the pod uses runtimeClassName: nvidia (section 3.0/3.4).
ai-gateway-controller crashloops (PostRouteModify SIGSEGV) after deleting a model
ERROR controller.inference-pool failed to sync InferencePool
{"error": "ExtensionReference service <model>-epp not found ..."}
panic: runtime error: invalid memory address or nil pointer dereference
... extensionserver.(*Server).PostRouteModify ...
Deleting an LLMModel removes its vLLM and EPP Deployments/Services,
but does not garbage-collect the per-model gateway resources
(InferencePool, AIGatewayRoute, HTTPRoute) - they have no
owner-ref back to the CR. The orphaned InferencePool keeps an
ExtensionReference to the now-deleted <model>-epp Service, and the
AI Gateway controller nil-derefs on it and crashloops. Delete the
leftovers by hand, then restart the controller:
