Public Reference Deployment¶

This guide documents the project's public reference deployment at platform.mcpruntime.org. To try its user and team workflows, start with the Public Platform Walkthrough. The operations below explain how its infrastructure is configured and maintained, and provide a worked example for operators building their own deployment.

The Runtime cluster uses K3s; the external identity provider runs on a separate VM with Docker and Caddy. K3s is the distribution selected for this example, not a requirement for MCP Runtime. See Deployment Options for other Kubernetes distributions and Self-Hosting for the general installation path.

This guide covers configuration, installation and updates, identity-provider integration, backups and recovery, and verification. Commands use the reference deployment's source checkout, env profile, and helper scripts. Adapt the domain, node architecture, SSH hosts, kubeconfig, storage, ingress, and identity provider to your own deployment. The reference scripts include K3s-specific backup and ingress behavior; do not apply them unchanged to a managed cluster.

Topology¶

Component	Reference placement	Public endpoint / responsibility
Runtime operator, APIs, UI, MCP servers, and telemetry	Kubernetes cluster using K3s	`platform.mcpruntime.org` and `mcp.mcpruntime.org`
Registry, Traefik, and cert-manager	Runtime cluster	`registry.mcpruntime.org`; Runtime ingress and TLS
Optional MCP authorization server (`mcp-auth`)	Runtime cluster, separate release track	`auth.mcpruntime.org/mcp-auth`; MCP OAuth issuance
External identity provider (Keycloak)	Docker service on the Buddy VM	`keycloak.mcpruntime.org`; users, realm data, and upstream login
Identity-provider HTTPS endpoint (Caddy)	Buddy VM, beside Keycloak	Keycloak TLS termination and certificate renewal

Cluster size, node names, and addresses can change. Inspect the selected kubeconfig with kubectl get nodes; the multi-node layout in Cluster Provisioning is a reference design, not a statement of the live cluster's node count. Keycloak and Caddy have their own backup and recovery boundary outside Kubernetes.

Obtain and select cluster access¶

For a provider-managed cluster, use the provider's supported login/configure command to write a kubeconfig context; see the per-distribution overview in Deployment Options. For this self-managed k3s cluster, use the isolated production file $HOME/.kube/prod-mcp-runtime-config when it is provisioned. Keep the default ~/.kube/config on the contributor test context and do not merge production credentials into it. If the production file is missing, an authorized operator can copy the k3s server kubeconfig from the control-plane VM:

source config/deployments/mcpruntime-org.env
install -d -m 700 "$HOME/.kube"
PROD_KUBECONFIG="$HOME/.kube/prod-mcp-runtime-config"
scp "root@${MCP_PRODUCTION_SSH_HOST}:/etc/rancher/k3s/k3s.yaml" "$PROD_KUBECONFIG"
chmod 600 "$PROD_KUBECONFIG"

# Use the control-plane API address reachable from this workstation. This
# changes only the endpoint; keep certificate-authority-data and TLS checks.
CLUSTER_NAME="$(kubectl --kubeconfig "$PROD_KUBECONFIG" config view --minify \
  -o jsonpath='{.clusters[0].name}')"
kubectl --kubeconfig "$PROD_KUBECONFIG" config set-cluster "$CLUSTER_NAME" \
  --server="https://<reachable-control-plane-address>:6443"

If SSH targets a worker or the control-plane API is not reachable from the workstation, obtain the supported API address/network path and a kubeconfig from the cluster operator. Do not disable TLS verification. Keep this file outside the repository and private (chmod 600).

Select and confirm the context before deploying. Use the isolated production kubeconfig and explicit context:

PROD_KUBECONFIG="${PROD_KUBECONFIG:-$HOME/.kube/prod-mcp-runtime-config}"
kubectl --kubeconfig "$PROD_KUBECONFIG" config get-contexts
kubectl --kubeconfig "$PROD_KUBECONFIG" --context prod-mcp-runtime get nodes

Prerequisites¶

# Keep the default kubeconfig on test; use the production file explicitly.
PROD_KUBECONFIG="$HOME/.kube/prod-mcp-runtime-config"
kubectl --kubeconfig "$PROD_KUBECONFIG" --context prod-mcp-runtime get nodes

# Build the current CLI from the selected Runtime ref.
make build
./bin/mcp-runtime --version

Production setup commands use the selected PROD_KUBECONFIG explicitly; do not export it as the workstation's default kubeconfig.

Configuration¶

Saved deployment profile (committed template + local override):

cp config/deployments/mcpruntime-org.env.example config/deployments/mcpruntime-org.env
# edit mcpruntime-org.env, see the Environment variable reference below

config/deployments/mcpruntime-org.env is gitignored. The .example file is the team-shared template; your local .env holds workstation-specific paths.

The hack scripts under hack/deploy/reference/ source this file by default. Override the path with MCP_DEPLOY_ENV=/path/to/other.env. See hack/README.md for the full layout.

Environment variable reference¶

Cluster access and domain¶

Variable	Required	Used by	Purpose
`KUBECONFIG`	yes	all hack scripts, manual `kubectl`	Path to the selected cluster kubeconfig. Must match the cluster you target.
`MCP_KUBE_CONTEXT`	optional	CLI setup and public hack scripts	Context within the kubeconfig; use when the file has multiple clusters.
`MCP_SETUP_KUBECONFIG`	yes (setup)	`mcp-runtime setup --kubeconfig`	Same as `KUBECONFIG`; passed to `mcp-runtime setup --kubeconfig`.
`MCP_PLATFORM_DOMAIN`	yes	setup	Apex domain only (no `https://`). Derives `registry.`, `mcp.`, and `platform.` hostnames.
`MCP_PLATFORM_ADMIN_EMAIL`	yes (non-test setup)	setup	Seeds the platform admin account during bootstrap.

Image build and registry pulls¶

Variable	Required	Used by	Purpose
`MCP_IMAGE_PLATFORM`	strongly recommended	setup, rollout	Target OS/arch for platform images (for example `linux/amd64` when nodes are amd64).
`MCP_REGISTRY_ENDPOINT`	yes (`bundled-https`)	setup, rollout (via configmap patch)	Hostname nodes use to pull platform and tenant images. With public TLS, set to `registry.<domain>`; do not use the registry Service ClusterIP.
`MCP_REGISTRY_INGRESS_HOST`	optional	rollout, CLI build/push	Public registry hostname for `docker push` / `server push`. Defaults from `MCP_PLATFORM_DOMAIN` when unset.
`MCP_REGISTRY_HOST`	do not set	—	Public ingress hostname; derived from `MCP_PLATFORM_DOMAIN`. Do not use as the internal pull URL.
`MCP_REGISTRY_INTERNAL`	optional	rollout	Override registry ClusterIP:port for build/push inside rollout script only. Pull path still uses `MCP_REGISTRY_ENDPOINT` in configmap.
`MCP_REGISTRY_PUSH_MODE`	`internal`	rollout	`public` pushes directly to `registry.<domain>` using the workstation's selected Docker daemon.

MCP Auth updates use a separately reviewed release manifest and mcp-runtime update --only mcp-auth. Runtime updates do not build, copy or change MCP Auth images.

Setup behavior (read by `mcp-runtime setup --env-file`)¶

Variable	Default	Purpose
`MCP_SETUP_WAIT_TIMEOUT`	`900`	Seconds to wait for setup rollouts.
`MCP_CERT_TIMEOUT`	`5m` (CLI default)	Certificate issuance wait on first install. Use `15m` on fresh clusters.
`MCP_SETUP_PLATFORM_MODE`	`public` (reference profile)	Passed to `setup --platform-mode`; the CLI default is `org`.
`MCP_SETUP_REGISTRY_MODE`	`bundled-https`	Passed to `setup --registry-mode`.
`MCP_SETUP_INGRESS`	`none`	`none` when k3s Traefik in `kube-system` already serves ingress.
`MCP_SETUP_TLS_CLUSTER_ISSUER`	`letsencrypt-prod`	ClusterIssuer name on reruns. Do not pass `--acme-email` when this issuer already exists.
`MCP_SETUP_SKIP_CERT_MANAGER_INSTALL`	unset	Set to `1` when cert-manager is already installed (typical reruns).

k3s Traefik integration (written to `mcp-shared-config`)¶

Variable	Typical value	Purpose
`PLATFORM_TRAEFIK_NAMESPACE`	`kube-system`	Namespace of the live Traefik deployment on k3s.
`PLATFORM_TEAM_TRAEFIK_WATCH`	`disabled`	Prevents `team create` from patching repo-managed `traefik/traefik` when k3s Traefik is external.

Variable	Required	Purpose
`GOOGLE_CLIENT_ID`	yes (public TLS)	Google OAuth client for dashboard sign-in.
`MCP_GOOGLE_CLIENT_ID`	optional	Alias for `GOOGLE_CLIENT_ID`.
`OIDC_ISSUER`	optional	Non-Google provider; setup fills Google defaults when `GOOGLE_CLIENT_ID` is set.
`OIDC_AUDIENCE`	optional	OIDC audience; defaults to Google client ID.
`OIDC_JWKS_URL`	optional	JWKS URL for token validation; discovered from `OIDC_ISSUER` when omitted and the provider publishes OIDC metadata.

MCP OAuth authorization server¶

Applies to MCP servers with spec.auth. These variables configure the authorization server that MCP clients use. Browser sign-in above configures OIDC for the dashboard.

Variable	Required	Purpose
`MCP_SETUP_MCP_AUTH_ISSUER_URL`	optional	Public HTTPS issuer URL; defaults to `https://auth.<MCP_PLATFORM_DOMAIN>/mcp-auth`.
`MCP_SETUP_MCP_AUTH_RESOURCE_URL`	optional	Initial canonical MCP resource URL; any supplied value must exactly match a protected server's `spec.auth.audience`. The operator reconciles the accepted list from current OAuth MCPServers.
`MCP_SETUP_MCP_AUTH_CONNECTORS_FILE`	required when enabled	Provider-neutral connector JSON; client secrets are referenced by environment variable, never stored in this file.
`MCP_SETUP_MCP_AUTH_CONNECTOR`	required when enabled	Named connector selected by the mcp-auth server.
`MCP_SETUP_MCP_AUTH_TLS_SECRET`	optional	Override the managed TLS Secret for the authorization-server hostname; required only with `--provided-tls-secrets`.
`MCP_SETUP_MCP_AUTH_SIGNING_KEY_SECRET`	required when enabled	Persistent RSA signing-key Secret containing `private-key.pem`.

The issuer must be the exact public URL configured for the optional mcp-auth-server, normally https://auth.<domain>/mcp-auth. This authorization-server hostname is separate from dashboard OIDC and from the Runtime gateway. The identity provider hostname, realm, client, users, redirect URI, scopes, and certificates are operated by the platform user.

Discovery is served at the authorization-server metadata URL:

curl -s https://auth.<domain>/.well-known/oauth-authorization-server/mcp-auth

The protected MCP resource separately publishes Protected Resource Metadata; clients should follow its WWW-Authenticate challenge or query the resource metadata URL generated for that server.

A ready-to-adapt protected server definition is in examples/oauth-example-go-2025-11-25/.mcp/servers.yaml. Its auth.issuerURL defaults from the bundled issuer configured during setup. auth.audience must be the server's canonical resource URI (https://mcp.<domain>/<prefix>/mcp). The gateway fails closed with 401 when a token's aud does not match.

Rollout-only (`mcp-runtime update`)¶

Use mcp-runtime update with the selected release or reviewed candidate manifest, an explicit --kubeconfig, --context, and --expect-cluster-id. For --build, use a clean source checkout matching the image source commit. Review --dry-run before applying with --yes. The CLI owns build, apply, rollout and rollback; no rollout shell wrapper is needed. It does not rewrite pull Secrets or tenant NetworkPolicies.

Multitenancy test (`hack/qa/multitenancy-test.sh`)¶

These are not in the deployment profile. Export them only for loopback or disposable .e2e.mcpruntime.org URLs; this provisioning demo refuses production hosts.

Variable	Example	Purpose
`PLATFORM_URL`	`http://127.0.0.1:18080`	Platform API base (no trailing slash); loopback or disposable staging only.
`MCP_URL`	`http://127.0.0.1:18080`	MCP ingress base on the same disposable target.
`REGISTRY_HOST`	`localhost:5000`	Registry hostname for disposable tenant image build/push.
`ADMIN_EMAIL` / `ADMIN_PASSWORD`	test admin creds	Platform admin login when not using token.
`ADMIN_TOKEN`	optional	Admin API token instead of password login.
`VERIFY_DEPLOY_PULL_SECRET`	`0`	When `1`, retain the supplied kubeconfig for read-only checks that `server deploy` created the namespace-local image pull Secret and attached it to `mcp-workload`.

The test script clears KUBECONFIG for CLI tenant flows, so server build, server push, and server deploy exercise the platform API without admin cluster access. To also assert the k3s namespace pull Secret after each deploy, set VERIFY_DEPLOY_PULL_SECRET=1 and provide the production KUBECONFIG; the script uses it only for those read-only Secret and ServiceAccount checks.

Do not set on this TLS production cluster¶

Variable	Why
`MCP_REGISTRY_ENDPOINT=10.x.x.x:5000`	ClusterIP breaks `bundled-https` TLS cert validation on pod pulls.
`MCP_ACME_EMAIL` on reruns	Re-applies Let's Encrypt issuer and can trigger duplicate-cert rate limits. Use `MCP_SETUP_TLS_CLUSTER_ISSUER` instead.
`MCP_RUNTIME_TEST_MODE=1`	Dev/test-mode guardrails; omit for production-shaped installs.

Minimal profile example¶

PROD_KUBECONFIG="$HOME/.kube/prod-mcp-runtime-config"
export MCP_PLATFORM_DOMAIN=mcpruntime.org
export MCP_IMAGE_PLATFORM=linux/amd64
export MCP_PLATFORM_ADMIN_EMAIL=admin@example.com
export MCP_REGISTRY_ENDPOINT=registry.mcpruntime.org
export GOOGLE_CLIENT_ID=<google-oauth-client-id>

See config/deployments/mcpruntime-org.env.example for the full saved profile used by the hack scripts.

Install and update¶

For an existing install, read Backups and recovery before a setup rerun or cleanup. First-time installs can proceed below.

First install (creates Let's Encrypt ClusterIssuer and certificates)¶

cp config/deployments/mcpruntime-org.env.example config/deployments/mcpruntime-org.env
# add GOOGLE_CLIENT_ID to mcpruntime-org.env when browser sign-in is required

export MCP_PLATFORM_ADMIN_EMAIL=admin@example.com

PROD_KUBECONFIG="$HOME/.kube/prod-mcp-runtime-config"
MCP_SETUP_WAIT_TIMEOUT=900 MCP_CERT_TIMEOUT=15m \
./bin/mcp-runtime setup \
  --env-file config/deployments/mcpruntime-org.env \
  --kubeconfig "$PROD_KUBECONFIG" \
  --with-tls \
  --acme-email ops@example.com \
  --ingress none \
  --registry-mode bundled-https \
  --platform-mode public

Reruns / upgrades (reuse existing certs to avoid LE rate limits)¶

Use an explicit private deployment profile and a clean checkout at the selected published release. Record the cluster UID read-only; do not select production as the ambient kube context. Configure these helper inputs in the profile:

export MCP_SETUP_KUBECONFIG=/path/to/prod-kubeconfig
export MCP_KUBE_CONTEXT=prod-mcp-runtime
export MCP_EXPECTED_CLUSTER_UID=<kube-system-namespace-uid>
export MCP_DEPLOY_REF=<selected-stable-release>

For ordinary release updates, from a clean checkout of the release tag with the production kubeconfig passed explicitly (this is what the hosted release pipeline runs):

make build VERSION=<tag> COMMIT=<tag-commit>
./bin/mcp-runtime update --to <tag> --build --source . --image-platform linux/amd64 \
  --kubeconfig "$MCP_SETUP_KUBECONFIG" --context "$MCP_KUBE_CONTEXT" \
  --expect-cluster-id "$MCP_EXPECTED_CLUSTER_UID" --dry-run
# review the plan, then repeat with --yes

Besides images, update applies the ServiceAccounts, RBAC, ConfigMaps and Deployment wiring (ServiceAccount, token mount, env, container flags, DNS options) the release declares in config/release/wiring.yaml, and restarts the workloads that read a changed ConfigMap, so such a release needs no full setup. --expect-cluster-id refuses a renamed or wrong context before anything is read or written. The dry-run plan lists every object, ConfigMap and wiring change; review it before --yes.

Operator ServiceAccount, ClusterRole and namespace-scoped roles/bindings are bundled from the target release's canonical config/rbac/ and k8s/ manifests. update corrects their drift before any image rollout, using server-side apply with setup's mcp-runtime field manager; a permission apply failure stops rollouts. The bundled MCP Auth Role is included only when its Deployment exists. --only skips RBAC and emits a warning, so use an unscoped update for permission changes. Both text and JSON dry runs preview the affected objects. RBAC changes are not rolled back with images. No separate manual RBAC step is needed.

After updating, run mcp-runtime cluster doctor: it checks effective operator ServiceAccount permissions against all bundled rules through SubjectAccessReview, including managed-server writes and the optional bundled issuer, and fails on recent forbidden informer errors. The caller needs permission to create SubjectAccessReviews. The operator manifest intentionally declares two replicas with --leader-elect; two ready pods can be normal high availability. Check Deployment/ReplicaSet rollout state and leader election before treating a second pod as a stuck rollout; the historical pod count alone cannot establish that.

update uses the published component manifest. All changed Runtime components roll sequentially with the CLI's rollback behavior. Existing registry-scoped pull Secrets, certificates, unmanaged configuration keys and MCP Auth remain unchanged. Supply existing Docker registry credentials through an approved login workflow; update never copies an admin key into Kubernetes pull Secrets.

Full setup is separate and must be deliberate:

hack/deploy/reference/backup.sh --setup
./bin/mcp-runtime setup --env-file "$MCP_DEPLOY_ENV" \
  --kubeconfig "$MCP_SETUP_KUBECONFIG" --context "$MCP_KUBE_CONTEXT" \
  --expect-cluster-id "$MCP_EXPECTED_CLUSTER_UID" \
  --strict-prod --with-tls --provided-tls-secrets --skip-cert-manager-install \
  --registry-mode bundled-https --platform-mode public --ingress none

Setup requires existing registry/platform TLS Secrets with these flags. Its preflight refuses a cluster whose ID differs from --expect-cluster-id and refuses to drop or swap an installed MCP Auth server: select it with --with-mcp-auth-server and its current image. Setup does not automatically restore backups. First installations that require certificate issuance use mcp-runtime setup with the reviewed issuance flags. MCP Auth image updates are a separate release track using an explicit component manifest and mcp-runtime update --only mcp-auth; do not rebuild a sibling repo as a side effect of a Runtime release.

Before a breaking release reset, follow the backup, fresh-install and recovery procedure below, including disposable rehearsal and database-native recovery. No wrapper authorizes data reset as part of an ordinary update.

Changing platform mode¶

The v0.9.0 Runtime configuration supports org and public platform modes; tenant remains a team publish scope. Before upgrading an older profile that uses MCP_SETUP_PLATFORM_MODE=tenant, explicitly choose org or public. The hosted reference platform uses public, with public previews/listings, signed-in organization publishing, and team publishing all available.

Take the protected setup snapshot described in Backups and recovery and build the CLI from the exact reviewed revision. Use the reviewed candidate manifest with --platform-mode:

./bin/mcp-runtime update --release-manifest /protected/candidate.json \
  --platform-mode public --kubeconfig "$MCP_SETUP_KUBECONFIG" \
  --context "$MCP_KUBE_CONTEXT" --expect-cluster-id "$MCP_EXPECTED_CLUSTER_UID" \
  --dry-run
# Review the plan, then repeat with --yes instead of --dry-run.

The CLI sets only PLATFORM_MODE in the existing shared ConfigMaps, prepares the mode's catalog namespaces, and restarts runtime-api, platform-api and UI. The candidate manifest controls image changes; missing mode-dependent entries keep their current images. There is no need to write ConfigMap/restart entries manually. Do not pass --only: it skips shared configuration and conflicts with --platform-mode. Existing namespaces with another scope or no catalog ownership are rejected. Newly created catalogs remain after rollback; they are never deleted by update. Record MCP_SETUP_PLATFORM_MODE=public in the private installation profile so later setup uses the same reviewed mode.

The first server deployment into a catalog prepares its namespace-local operator permissions, registry pulls, quotas and network protection for admins and users. Creating only a namespace is insufficient to host a server.

This transition needs no fresh setup or data reset. Preserve platform databases, registry images, Secrets, certificates, and the separately hosted identity provider's data/TLS state. Retain the existing mcp-auth release. Keep the snapshot and prior exact image references outside git. If verification fails, recover the previous three images and prior ConfigMap mode through a reviewed rollback update manifest; restore only those changed fields.

Verify signed-in login, CLI image publishing in all enabled scopes, server deployment, anonymous public-catalog visibility, and anonymous denial for org and team catalogs. Run hosted certificate/grant acceptance for allowed and denied calls as well. The test-mcpsnoop example provides a separate unauthenticated two-tool server for payload inspection.

Upgrading an existing cert-manager installation¶

setup retains an existing cert-manager installation; the new install pin does not perform a multi-minor upgrade. cluster doctor checks the controller, webhook, and cainjector versions against the reviewed upstream support matrix. It reports retired, mismatched, unknown, or unsupported versions rather than counting Ready pods as compatibility evidence.

For the old 1.16 installation, rehearse upgrades one minor at a time through 1.17, 1.18, 1.19, 1.20, and 1.21 using the latest patch in each series. Read each minor's upgrade notes and use the same installation method throughout. Before starting, make an encrypted off-host backup of cert-manager resources, CA/private-key Secrets, ACME account Secrets, and public/internal TLS Secrets. Record certificate fingerprints, issuer readiness, and renewal times. Never uninstall the CRDs, delete signing material, or force leaf reissuance to update the controller. Preserve any custom approval controller configuration.

Upgrade notes between 1.16 and 1.21 that affect MCP Runtime (reviewed 2026-10-06 against the upstream release notes for each minor):

Minor	Change	Effect on MCP Runtime
1.17	CA and SelfSigned issuers sign with SHA-384/512 for RSA keys of 3072/4096 bits or more; unstructured logs replaced by structured logs.	The bundled `mcp-runtime-ca` is ECDSA P-256, so this does not apply to it. Check this only for an operator-supplied RSA CA of 3072 bits or more. Update any log grep in alerts.
1.18	`Certificate.spec.privateKey.rotationPolicy` defaults to `Always`; `revisionHistoryLimit` defaults to `1`; the ACME HTTP-01 solver Ingress uses `pathType: Exact`.	Leaf keys rotate on the next renewal, which is the intended behavior. `mcp-runtime-ca` is a setup-generated Secret, not a Certificate, so its key does not rotate. Old CertificateRequests are garbage-collected on upgrade. Traefik accepts `Exact`; only ingress-nginx is affected.
1.19	The ACME client metrics drop their `path` label in favor of `action`; `CAInjectorMerging` is on by default. Skip v1.19.0, which reissues certificates unnecessarily.	Update dashboards that use the old label.
1.20	Containers run as UID/GID 65532 instead of 1000/0; the `cert-manager-edit` ClusterRole no longer creates Challenges or Orders; `DefaultPrivateKeyRotationPolicyAlways` is GA.	MCP Runtime does not create Challenges or Orders directly.
1.21	`CAInjectorMerging` is GA; the controller metrics port is renamed `http-metrics`; Helm drops the default `tokenrequest` RBAC and three metrics values.	The static manifest that setup applies has no Helm values. Update scrape configs that select the old port name.

None of these minors removes a CRD API version that MCP Runtime uses (cert-manager.io/v1) or a flag that setup passes. Setup applies the upstream static manifest unchanged.

After each step on disposable Staging E2E, wait for controller, webhook, and cainjector readiness; verify existing issuers and Certificates remain Ready; issue a fresh bounded test CertificateRequest; and check renewal scheduling and unchanged CA/TLS identities. Stop on failure and recover using the documented upstream rollback procedure and the backed-up configuration. Only then schedule the production upgrade and repeat the read-only inventory checks.

Reviewed 2026-10-03: supported releases and upgrade guidance. The compatibility check currently recognizes supported 1.20 and 1.21 series; update that reviewed table when adopting a new upstream minor.

Identity provider¶

The reference deployment uses Keycloak as its external identity provider and runs the optional MCP authorization server inside the Runtime cluster. Other deployments can use a compatible OIDC provider; a separate Keycloak VM and Caddy are choices made by this reference, not Runtime prerequisites.

Optional bundled MCP authorization server¶

OAuth-enabled MCP Runtime servers require bearer tokens, Protected Resource Metadata, and resource audience validation. The bundled mcp-auth-server is optional; when enabled it authenticates users through one external OIDC identity provider such as Keycloak and issues MCP access tokens. Runtime governance decisions stay in the gateway.

The Runtime gateway is the protected-resource boundary. It verifies the issuer, signature, audience/resource, expiry, and scope, then applies grants, agent sessions, trust, and tool policy. The authorization server sees only login and token requests; Runtime policy needs the MCP JSON-RPC tool call and current grant/session state, so the two stay separate.

For the reference deployment, the two authentication hosts have different locations. Point auth.<domain> at Runtime ingress and keycloak.<domain> at the identity-provider VM; keep the provider's advertised HTTPS endpoints consistent with that DNS choice:

auth.<domain>      -> <Runtime ingress IP>
keycloak.<domain>  -> <identity-provider VM IP>

Deploy Keycloak with a realm such as mcp-runtime and a confidential client named mcp-auth. Configure this exact upstream callback URI:

https://auth.<domain>/mcp-auth/identity/callback

For the mcpruntime.org demo, Keycloak is a separate Docker service on the Buddy VM (devbox1, 103.181.176.61). Its Compose file mounts the retained realm database at /opt/keycloak/data and caps Keycloak at 900 MiB on this 2 GiB VM. This H2 configuration is for a single-instance demo. The existing Caddy instance on that VM serves keycloak.mcpruntime.org using the Caddy site and obtains and renews its Let's Encrypt certificate. The DNS A record for keycloak.mcpruntime.org must point to 103.181.176.61.

Runtime setup does not deploy, back up, or restore this identity provider. Back up /opt/keycloak/data while the Keycloak container is stopped, and keep the client secret outside the Compose and Caddy files. Start the service with docker compose -f /opt/keycloak/compose.yaml up -d on devbox1; append the Caddy site to /opt/workspace/Caddyfile, validate it with caddy validate, and reload Caddy. Keep the existing workspace.mcpruntime.org site in place. The mcp-auth connector should use the public HTTPS token and JWKS endpoints rather than in-cluster Service addresses. A direct public discovery check proves Keycloak is reachable; verify an authorization-code callback as well to prove mcp-auth can exchange a code.

curl -fsS https://keycloak.mcpruntime.org/realms/mcp-runtime/.well-known/openid-configuration
curl -fsS https://auth.mcpruntime.org/.well-known/oauth-authorization-server/mcp-auth
ssh devbox1 'docker ps --filter name=demo-keycloak; docker logs demo-keycloak --tail 30'

For the mcpruntime.org demo, use the tracked connector file and set MCP_SETUP_MCP_AUTH_CONNECTORS_FILE=config/deployments/mcpruntime-org-keycloak-connectors.json in the private deployment env file. It references the Keycloak issuer and client but never stores the client secret. The relevant fields are:

{
  "keycloak": {
    "issuer": "https://keycloak.<domain>/realms/mcp-runtime",
    "authorization_endpoint": "https://keycloak.<domain>/realms/mcp-runtime/protocol/openid-connect/auth",
    "token_endpoint": "https://keycloak.<domain>/realms/mcp-runtime/protocol/openid-connect/token",
    "jwks_uri": "https://keycloak.<domain>/realms/mcp-runtime/protocol/openid-connect/certs",
    "client_id": "mcp-auth",
    "client_secret_env": "KEYCLOAK_CLIENT_SECRET",
    "exchange_client_id": "mcp-auth",
    "scopes": ["openid", "profile", "email"],
    "identity_claims": ["preferred_username"],
    "token_endpoint_auth_method": "client_secret_post",
    "allowed_upstream_callback_uris": [
      "https://auth.<domain>/mcp-auth/identity/callback"
    ],
    "downstream_token_strategy": "upstream_session"
  }
}

For an existing installation, update the identity connector without changing its credential Secret or other platform configuration:

./bin/mcp-runtime cluster mcp-auth-connector apply \
  --file config/deployments/mcpruntime-org-keycloak-connectors.json \
  --connector keycloak --kubeconfig "$PROD_KUBECONFIG" \
  --context "$PROD_CONTEXT" --dry-run
./bin/mcp-runtime cluster mcp-auth-connector apply \
  --file config/deployments/mcpruntime-org-keycloak-connectors.json \
  --connector keycloak --kubeconfig "$PROD_KUBECONFIG" \
  --context "$PROD_CONTEXT" --yes

The next MCP Auth rollout loads the updated file. Connector configuration selects the identity provider; MCP scope policy belongs in each server's auth.scopes, which the operator passes to MCP Auth per resource.

Deploy the optional bundled server through normal setup:

./bin/mcp-runtime setup \
  --with-tls --tls-cluster-issuer letsencrypt-prod \
  --with-mcp-auth-server \
  --mcp-auth-signing-key-secret mcp-auth-signing-key \
  --mcp-auth-connectors-file /secure/mcp-auth-connectors.json \
  --mcp-auth-connector keycloak

The issuer defaults to https://auth.<MCP_PLATFORM_DOMAIN>/mcp-auth, and the operator derives auth.issuerURL and reconciles accepted resource audiences from OAuth MCPServers. Any optional --mcp-auth-resource-url must exactly match an MCPServer's spec.auth.audience. Production requires a certificate covering the auth host (provisioned by the configured TLS ClusterIssuer), a selected connector, and a persistent RSA signing key stored in the Secret key private-key.pem. Use --mcp-auth-tls-secret only for an externally managed certificate. The connector's KEYCLOAK_CLIENT_SECRET value is read from the environment and converted into a Kubernetes Secret; it must not be committed to Git.

Setup waits for the current auth and operator Deployment revisions to finish rolling out. A healthy old auth pod does not prove the new image or resource allowlist is active. If authorization returns resource is not recognized, compare MCP_AUTH_RESOURCES on the serving pods with the Deployment and inspect replacement pod startup logs. A published image that rejects connector fields must be replaced with a compatible image; preserve the connector config, signing key, and data PVC during recovery. cluster diagnostics checks the current auth revision, observed generation, and replacement replicas; API access failures are reported rather than treated as an absent optional auth installation.

The bundled server uses SQLite on a PVC in production and memory storage only in --test-mode. Test mode also permits the loopback development issuer and an ephemeral signing key. A public deployment must use HTTPS for Keycloak's issuer, authorization endpoint, token endpoint, and JWKS endpoint. Use internal HTTP only for local testing.

Backups and recovery¶

Back up the Runtime cluster and the identity-provider VM separately. Runtime backup scripts do not include Keycloak realm data or Caddy state; see Identity provider for that boundary. Before reinstalling an existing platform, review the backup coverage and recovery steps below.

v0.10.0 session-storage release¶

The source/install revision is v0.10.0. Use its CLI and manifests together; policy schema v4 does not support mixed older gateways. The new platform sync listener, Pod-bound token mounts, RBAC and network policy require fresh setup for this breaking release, followed by supported CLI server deployment.

Before an authorized deployment, record the old and new commit IDs, cluster UID, and a protected copy of the existing deployment profile, including platform mode, domains, ingress, storage, registry, OIDC and whether mcp-auth is installed. Use the same reviewed configuration for setup from v0.10.0; retain the separately versioned mcp-auth choice. Do not substitute example credentials.

Complete the full backup below plus database-native exports for Postgres, ClickHouse and any other persisted application stores. Record the immutable snapshot/export locations and checksums in the private deployment record; copy them to protected off-host storage. Preserve registry image contents, TLS/CA keys, pull credentials and required platform Secrets. Separately snapshot Keycloak data and Caddy configuration/certificate state.
Rehearse fresh setup and recovery on disposable infrastructure with the recorded configuration. Restore required users, teams, agents, grants and application data. Kubernetes session objects are recovery evidence only: this release intentionally does not import them into the new session model. If restoring a v0.10.0 Postgres database, retain session identity reservations, revocation records and retirement fences alongside session rows.
After recovery is proven and the reset is explicitly authorized, use the guarded reset procedure below, then the installation commands in this guide from v0.10.0 with the recorded profile. Restore required database/registry contents and selected configuration/certificates before accepting traffic. The configuration restore helper does not restore database contents.
Restart platform-api after every Postgres restore, before gateway reconnection, to rotate the sync epoch. Redeploy servers through the CLI to install matching gateway images, tokens and egress rules. Re-enroll adapters to issue new session names and certificates; do not reuse deleted session names.
Verify login, registry publish and fresh pulls, server deployment, enrollment, grant/session creation, allowed and denied calls, revocation and cross-team isolation. Check gateway sync freshness and correlated audit/log/trace data.

If recovery or acceptance fails, keep traffic closed and recover the prior recorded release with its matched databases, configuration and certificates from the rehearsed backup. Do not reconnect old gateways to schema v4 or copy old Kubernetes sessions into the new store. This release procedure does not itself authorize a reset or deployment of the hosted reference platform.

Production backups¶

Before a production setup redeployment, capture resources and credentials that setup may reapply. The snapshot includes PVC/PV definitions, but does not copy live volume contents because setup does not delete claims or their data:

hack/deploy/reference/backup.sh --setup

The command requires the isolated prod-mcp-runtime context. It creates a timestamped snapshot under ~/.mcpruntime/backups/mcpruntime-org/ and updates latest only after all parts validate. The snapshot directory is mode 0700; files containing Secrets are mode 0600. Setup merges existing resources and reuses matching PVCs, but may update config/Secrets and roll workloads. An immutable StatefulSet change can recreate that StatefulSet while leaving its claims in place. The resource inventory is for recovery reference, not bulk kubectl apply; its selected platform files are consumed by the existing setup restore path.

On a fresh public TLS install, setup uses a bcrypt-hashed bootstrap credential in a Kubernetes Secret so registry authentication does not depend on a platform API that has not pulled its own image yet. The operator receives matching pull credentials before rollout. Once the API is ready, setup replaces bootstrap authentication with platform forward authentication and enables native tokens; the bootstrap middleware and hashed Secret are removed. An interrupted setup keeps an authenticated route and can be rerun. Existing native scoped pull credentials are preserved.

Full setup keeps public registry ingress authentication enabled while images are built and published through the internal registry helper. After native token authentication is enabled, setup reruns preserve repository-scoped registry access while retaining public ingress authentication, backend token settings, the token signer, and namespace-scoped pull configuration.

For destructive cleanup or full node recovery, create a full backup:

hack/deploy/reference/backup.sh --full --online-copy

The full snapshot also requires the configured SSH host and captures K3s control-plane state and local-path volume files without stopping workloads.

A full snapshot includes namespaced and cluster-scoped Kubernetes objects, CRDs, Secrets, grants, sessions, PV/PVC specs, a consistent SQLite online backup of the K3s control-plane database, /etc/rancher/k3s, K3s server credentials, and every file under /var/lib/rancher/k3s/storage. That volume archive includes Postgres, ClickHouse, Kafka, mcp-auth, observability, and registry data. It also contains the TLS and platform config/Secret files used by the existing setup restore path, plus a SHA-256 manifest.

The demo Keycloak database and Caddy certificate now live on devbox1, so this K3s backup does not cover them. Stop demo-keycloak for a consistent backup of /opt/keycloak/data; back up the Caddy config and /opt/workspace/caddy_data on that VM separately.

The bundle is not encrypted by the backup command; file permissions reduce local access but do not protect it from device loss or disk compromise. Encrypt it with the team's approved storage before copying it off-host.

The full backup runs without stopping workloads. Kubernetes objects and SQLite state are captured online; PVC files are copied live and are not guaranteed to be an application-consistent point-in-time image. Databases may need WAL recovery after restore. Store this bundle on encrypted storage and copy it off the production node and workstation. Full node recovery restores the K3s state and volume archive on the original node. Validate the SHA-256 manifest before restoring.

For a full node restore, provision the same host and K3s version, stop K3s, extract k3s-host-and-pv-data.tar.gz at /, copy k3s-state.db to /var/lib/rancher/k3s/server/db/state.db with owner root:root and mode 0600, then start K3s and validate node and workload readiness. Keep the existing server token and /etc/rancher/k3s configuration from the archive. Verify the bundle first with cd ~/.mcpruntime/backups/mcpruntime-org/latest && shasum -a 256 -c SHA256SUMS. Check each database's recovery logs before accepting traffic. This does not restore an external identity provider or data stored outside the cluster.

hack/deploy/reference/reset.sh still takes a smaller platform-runtime restore snapshot before its intentional namespace wipe. It covers platform TLS/config/bootstrap material, not the full node or PVC data.

Variable	Default	Purpose
`MCP_TLS_BACKUP_DIR`	`~/.mcpruntime/backups/mcpruntime-org`	Root directory for timestamped setup or full snapshots.
`MCP_EXPECTED_CLUSTER_UID`	required	Actual `kube-system` namespace UID; all reference helpers verify it.
`MCP_DEPLOY_ENV`	`config/deployments/mcpruntime-org.env`	Env file path for all hack scripts.

The reset.sh snapshot covers TLS, cert-manager, OIDC, and bootstrap secrets. Use backup.sh --setup before setup, or backup.sh --full --online-copy for the Kubernetes object and persistent volume recovery bundle described above.

Step 0: Back up platform-runtime state before any wipe¶

A reset needs a completed full backup from this cluster, checked file digests, and rehearsed application/database recovery. A TLS/setup snapshot does not preserve volume contents and cannot authorize a reset. Keep protected off-host copies and separately back up the external identity provider.

hack/deploy/reference/backup.sh --full --online-copy

The full helper captures K3s SQLite state consistently, but copies live PVC files. It does not replace database-native exports or prove they can be restored. Record a reviewed recovery snapshot path; do not rely on a changing latest link.

Configuration recovery is explicit and separate from setup:

hack/deploy/reference/restore.sh --snapshot /protected/recovery/snapshot --dry-run
hack/deploy/reference/restore.sh --snapshot /protected/recovery/snapshot --yes

This restores selected TLS/config/credentials and restarts APIs, not databases or PVC contents. Resource conflicts fail for review instead of force-overwriting another manager's state. Complete database recovery before accepting traffic.

Safe cluster wipe (app workloads only)¶

Only an intentionally authorized reset should use:

hack/deploy/reference/reset.sh --dry-run
hack/deploy/reference/reset.sh --yes \
  --confirm-cluster "$MCP_EXPECTED_CLUSTER_UID" \
  --recovery-snapshot /protected/recovery/full-snapshot

The helper checks context, cluster identity, snapshot identity, completion and checksums before deleting selected Runtime namespaces and their MCP resources. It preserves system/unrelated namespaces, traefik, and the profile-selected ingress namespace, waits for namespace deletion, and retains a current credential snapshot without replacing the full recovery bundle. Then follow the rehearsed fresh-setup and restoration steps explicitly.

If you accidentally wiped kube-system¶

If kube-system pods are gone (no CoreDNS, no Traefik), restart k3s on the control plane to trigger full reconciliation from /var/lib/rancher/k3s/server/manifests/:

ssh root@103.181.176.28 "systemctl restart k3s"
# Wait for CoreDNS, Traefik, and svclb pods to come up
kubectl wait pod -n kube-system \
  -l app.kubernetes.io/name=traefik \
  --for=condition=Ready --timeout=120s

Verify port 80 is reachable before running setup with TLS:

curl -sm5 http://registry.mcpruntime.org/ && echo "port 80 OK"
# Expected: "404 page not found" from Traefik

Verification¶

Post-setup check¶

./bin/mcp-runtime cluster diagnostics

# Confirm TLS certs are Ready
kubectl get certificate registry-cert -n registry
kubectl get certificate -n mcp-platform

# Check platform and telemetry pods
kubectl get pods -n mcp-platform
kubectl get pods -n mcp-observability

Expected: platform and telemetry pods 1/1 Running, certificate READY=True.

Separate release tracks and user verification¶

The Runtime CLI binaries and hosted platform images are separate artifacts. Publishing a stable Runtime release runs .github/workflows/release.yaml to publish the CLI binaries. An operator then manually starts hosted-platform-release-deploy.yaml with that published tag to update the hosted platform. The deploy workflow waits for the release manifest, prints a dry-run plan, builds and rolls changed Runtime images, and leaves Secrets and PVCs in place. It leaves MCP Auth on its current image unless an operator passes mcp_auth_version for a manual Auth upgrade. The workflow uses GitHub's production environment and updates the README's live-version badge only after platform health and a temporary qa-audit-* MCP server pass their post-deploy checks. Publishing a release never triggers the production deploy automatically.

Verify the user path after rollout:

Install the candidate CLI built from the selected Runtime ref and confirm mcp-runtime --version reports that commit. After release publication, verify the releases/latest binary download reports the release tag.
Follow Quickstart against https://platform.mcpruntime.org: log in, build/push/deploy a temporary qa-audit-* server, create a grant, and verify the adapter result matches the configured identity mode. The hosted preview currently leaves adapter identity verification off, so its allow-list call returns missing_identity. Verify a successful governed call separately on the guarded staging target with adapter certificates enabled.
Check the signed-out platform page and signed-in role-gated UI with browser evidence. Use only temporary qa-audit-* resources and clean them up.
Verify mcp-runtime status, server listing, deployment readiness, and cluster doctor; confirm the existing certificate resources remain Ready.

Users access the same https://platform.mcpruntime.org URL after a platform rollout. They update the CLI separately from the GitHub Releases page. Until a new tag is published, releases/latest still downloads the previously published CLI.

Tenant push and deploy smoke test¶

After setup, verify a non-admin team member can publish and deploy:

ADMIN_KEY="$(kubectl get secret mcp-platform-api-credentials -n mcp-platform \
  -o jsonpath='{.data.ADMIN_API_KEYS}' | base64 -d | cut -d, -f1)"

# Admin: create team + user
MCP_PLATFORM_API_URL=https://platform.mcpruntime.org MCP_PLATFORM_API_TOKEN="$ADMIN_KEY" \
  ./bin/mcp-runtime team create myteam --name "My Team"
MCP_PLATFORM_API_URL=https://platform.mcpruntime.org MCP_PLATFORM_API_TOKEN="$ADMIN_KEY" \
  ./bin/mcp-runtime team user create myteam \
  --email member@example.com --role member

# Team member: login, build, push, deploy from metadata
MCP_PLATFORM_API_URL=https://platform.mcpruntime.org \
  ./bin/mcp-runtime auth login --email member@example.com --password 'YourPassword123!' \
  --profile myteam-user

cd examples/oauth-example-go-2025-11-25
# .mcp/servers.yaml already exists in the example; for a new server run:
# ../../bin/mcp-runtime server init <name> --tool <tool> --metadata-dir .mcp

MCP_PLATFORM_API_URL=https://platform.mcpruntime.org MCP_PLATFORM_API_PROFILE=myteam-user \
  ../../bin/mcp-runtime server build image oauth-example-go-2025-11-25-gateway \
  --metadata-dir .mcp \
  --tag verify-e2e \
  --platform linux/amd64

IMAGE_REF="$(awk '$1=="image:"{i=$2} $1=="imageTag:"{t=$2} END{print i ":" t}' .mcp/servers.yaml)"

MCP_PLATFORM_API_URL=https://platform.mcpruntime.org MCP_PLATFORM_API_PROFILE=myteam-user \
  ../../bin/mcp-runtime server push --scope tenant --image "$IMAGE_REF"

MCP_PLATFORM_API_URL=https://platform.mcpruntime.org MCP_PLATFORM_API_PROFILE=myteam-user \
  ../../bin/mcp-runtime server deploy oauth-example-go-2025-11-25-gateway \
  --scope tenant \
  --metadata-dir .mcp

Expected: push succeeds in under ~30s; deploy reports status Ready; the team namespace contains mcp-runtime-registry-pull and a running MCPServer pod. This platform CLI path also verifies that the API provisions a namespace-local registry pull Secret and attaches it for workload pulls. Keep this server push and server deploy smoke in k3s production QA. kubectl apply alone bypasses the platform's namespace and pull-secret provisioning path. The .mcp metadata must contain tools[*].sideEffect; server deploy copies that metadata into the platform request so governed tools/call requests can authorize side effects.

If team create returns 500 failed to provision team namespace, confirm PLATFORM_TEAM_TRAEFIK_WATCH=disabled is present in mcp-shared-config (or set it in config/deployments/mcpruntime-org.env before rerunning setup).

Multi-tenancy end-to-end test¶

hack/qa/multitenancy-test.sh

Default assumptions: - PLATFORM_URL=https://platform.mcpruntime.org - MCP_URL=https://mcp.mcpruntime.org - REGISTRY_HOST=registry.mcpruntime.org (image build tagging and push target resolution) - Team owners publish images with server push --scope tenant (platform API), not admin registry push - Builds and deploys acme-tools, globex-tools, and techcorp-tools example servers - Creates Acme, Globex, and TechCorp teams, applies cross-tenant grants - Verifies adapter success, dashboard events, and no-kubeconfig smoke checks

To skip the build/deploy and only verify an existing setup:

SKIP_SETUP=1 hack/qa/multitenancy-test.sh

Production observability and debugging¶

Start incident investigation at production Grafana. Its provisioned data sources are Prometheus (metrics), Loki (logs), and Tempo (traces). Use the same UTC time range across all three; record the affected workload, deployment image, and request/trace IDs. Inspect the affected client's own logs too, particularly for OAuth and MCP transport failures.

Read private operator credentials from ~/.mcpruntime/infra.env: GRAFANA_URL identifies the endpoint, and the saved Secret entries ending in GRAFANA_ADMIN_USER__BASE64 and GRAFANA_ADMIN_PASSWORD__BASE64 contain Grafana credentials. Browser access also requires a signed-in platform admin session. API access needs an accepted platform admin x-api-key for the ingress gate as well as Grafana authentication. Keep credentials in memory; never print them or put them in command arguments, tickets, or Buddy notes.

Metrics: verify target health and freshness, request/error rates, latency, restarts, resource pressure, and collector/exporter failures. Confirm that the affected service actually has a scrape target.
Logs: discover the actual Loki labels and query auth, ingress/gateway, and MCP server workloads over the incident window. Verify collection from all relevant namespaces and containers. Preserve event timestamps, HTTP status, safe failure reason, request ID, and workload identity.
Traces: verify spans reach Tempo and follow the failure across instrumented services. Check service identity, operation, duration, error status, and trace/span IDs in related logs. Record missing instrumentation and propagation explicitly; a missing trace does not prove success.

Trace and log correlation. Platform services and the MCP gateway append trace_id=<32 hex> span_id=<16 hex> to request and failure log lines. Copy the trace_id from a Loki line into Tempo (or search Loki for a Tempo trace ID) to join them. Spans for 401, 403, and 5xx responses carry error status plus mcp.failure.operation, mcp.failure.reason, and http.response.status_code; in Tempo, search status=error. The gateway records the policy decision reason, RPC method, and status only, never tokens or tool arguments. Health, readiness, and metrics probes are not traced. Known boundaries: external OAuth authorization servers and MCP clients that do not send W3C traceparent start a new trace at the gateway, so a client-side refresh failure cannot be joined to the gateway trace by ID; correlate by time window, workload, and status instead. Collector export failures are logged as otel internal error: ... and counted by the mcp_otel_internal_errors_total metric on each service's metrics port; a non-zero rate means spans are being dropped before Tempo.

Distinguish missing instrumentation from broken collection or an incorrect query/time range. Verify improvements through a real request and its resulting telemetry. Use read-only pod logs or authorized datasource reads as a documented fallback if Grafana is unavailable. Never capture tokens, passwords, or MCP tool payloads merely to improve debugging.

For each concrete maintainability or debuggability gap, search existing repository issues before creating a ticket. Include redacted evidence, affected components, scope, and acceptance checks; attach the new or existing issue to Maintainability and Debuggability Improvement.

For a Grafana 401, distinguish the platform admin gate from Grafana's login. A configured GF_SECURITY_ADMIN_PASSWORD matching the saved Secret does not prove the persisted account accepts it. Before an authorized password recovery, back up the persistent database and inspect grafana cli admin reset-admin-password --help. Use --password-from-stdin, preserve dashboards/data sources, verify authenticated API access afterward, and save the working credential and URL in private infra.env. Do not reset a persisted account automatically during diagnosis.

Troubleshooting¶

TLS cert not issued after 5+ minutes¶

kubectl describe challenge -A: look for ACME HTTP-01 status
kubectl logs -n cert-manager deploy/cert-manager --tail=60
Check Traefik is serving port 80: curl -sm5 http://mcp.mcpruntime.org/
Verify DNS: dig registry.mcpruntime.org +short should return 103.181.177.16
If a stale Certificate owns registry/registry-tls, delete it before rerunning setup:
```
kubectl delete certificate registry-tls -n registry --ignore-not-found
```

Traefik 404 on the registry host / node pulls fail with `NotFound`¶

Symptom: every request to https://registry.<domain>/v2/..., including anonymous /v2/, returns Traefik's plain 404 page not found instead of 401, and new pods fail to pull registry.<domain>/... with NotFound. Check the registry Ingress:

kubectl get ingress registry -n registry \
  -o jsonpath='rules={.spec.rules[*].host} tls={.spec.tls[*].hosts}{"\n"}'
./bin/mcp-runtime cluster doctor   # "registry Ingress hosts" check

If rules=registry.local while tls=registry.<domain>, Traefik has no router for the public host. This happens when the non-TLS base manifest (config/registry/base, rule host registry.local) is applied over a public install, for example by a registry apply from a shell that lacks MCP_PLATFORM_DOMAIN, or by kubectl apply -f config/registry/base/ingress.yaml (the QA E2E cache refresh) against the wrong context. The apply replaces spec.rules but keeps the existing spec.tls. Setup now resolves the host from the live Ingress TLS host and mcp-shared-config, and refuses to apply a registry.local rule host over a public Ingress. Fix a live cluster with the command doctor prints:

kubectl patch ingress registry -n registry --type=json \
  -p '[{"op":"replace","path":"/spec/rules/0/host","value":"registry.<domain>"}]'
curl -s -o /dev/null -w '%{http_code}\n' https://registry.<domain>/v2/   # expect 401

Also confirm traefik.ingress.kubernetes.io/router.entrypoints is websecure; the base manifest sets web.

Setup fails "bundled registry platform setup requires MCP_REGISTRY_ENDPOINT"¶

You omitted --test-mode and used --registry-mode auto. For this k3s cluster use --registry-mode bundled-https (used by the production mcp-runtime setup flags above). Do not export a ClusterIP as MCP_REGISTRY_ENDPOINT on the public TLS deployment.

Setup fails "MCP_IMAGE_PLATFORM does not match Kubernetes node architecture"¶

Set MCP_IMAGE_PLATFORM=linux/amd64 (cluster nodes are amd64; local Mac is arm64).

kube-system empty / HelmChart CRD missing¶

See If you accidentally wiped kube-system above. Restart k3s on the control plane. Do not re-create the HelmChart CRDs manually.

Namespaces stuck in Terminating¶

for ns in $(kubectl get ns --no-headers | awk '$2=="Terminating"{print $1}'); do
  kubectl get ns "$ns" -o json \
    | jq '.spec.finalizers = []' \
    | kubectl replace --raw "/api/v1/namespaces/$ns/finalize" -f -
done

Let's Encrypt rate limit hit¶

Restore the backed-up TLS secrets (Step 0). Do not re-request certs:

kubectl apply -f /tmp/registry-tls-backup.yaml
kubectl apply -f /tmp/platform-tls-backup.yaml

Check current usage at https://crt.sh/?q=mcpruntime.org.

The disposable Kind QA harness now adds fresh certificate issuance to its adapter-certificate scenario. It reapplies the supported certificate setup, issues a one-hour test certificate, verifies its SAN, validity and workload CA chain, and compares public workload-CA/platform-TLS fingerprints before and afterward. The fixture never reads private keys. This covers fresh issuance and preservation during reapplication; the minor-by-minor upgrade and renewal procedure still needs a disposable staging run.