Skip to content

Contributor Troubleshooting

Start from the symptom and inspect the narrowest surface first.

Anonymous Users Can See MCPs

Expected behavior:

curl -i http://localhost:18080/api/runtime/servers

The response should be 401 Unauthorized.

If it returns servers:

  1. Confirm the UI and API Deployments are on the current images.
kubectl get deploy mcp-sentinel-api mcp-sentinel-ui -n mcp-sentinel \
  -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[0].image}{"\n"}{end}'
  1. Roll both Deployments after patching auth or catalog code.
kubectl rollout status deployment/mcp-sentinel-api -n mcp-sentinel --timeout=90s
kubectl rollout status deployment/mcp-sentinel-ui -n mcp-sentinel --timeout=90s
  1. Sign out in the browser or use a fresh private window. A valid UI session cookie makes the same URL authenticated.

Tenant User Sees the Wrong MCPs

Check the API path first:

curl -sS -b /tmp/mcp-tenant-a-cookie.txt \
  http://localhost:18080/api/runtime/servers |
  jq '[.servers[] | {namespace,name,team_id}]'

Then compare Kubernetes state:

kubectl get mcpservers -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TEAM:.spec.teamID'

Rules to verify:

  • Single-team/example MCPs in this guide live in mcp-servers.
  • Org-mode and public-mode catalogs live in mcp-servers-org and mcp-servers-public; tenant MCPs live in mcp-team-<slug> namespaces with matching ownership.
  • Non-admin users should receive 403 when explicitly requesting another tenant namespace.
  • Direct kubectl get mcpservers -A is not an authz check; it shows what your kubeconfig can read.

tools[].sideEffect Is Required

Every listed tool must declare a side effect:

tools:
  - name: add
    requiredTrust: low
    sideEffect: read

If the operator logs validation errors such as spec.tools[0].sideEffect: Required value, check for CRD/operator skew and old test objects:

kubectl apply -f config/crd/bases/mcpruntime.org_mcpservers.yaml
kubectl logs -n mcp-runtime deploy/mcp-runtime-operator-controller-manager --since=10m
kubectl get mcpservers -A -o yaml | rg -n "name:|sideEffect"

Patch old local test objects or redeploy them from current metadata.

Image Pulls Fail

For Kind test mode, pod images should use:

registry.registry.svc.cluster.local:5000/<image>:<tag>

The Kind cluster must have a containerd mirror for that exact host:

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.registry.svc.cluster.local:5000"]
  endpoint = ["http://127.0.0.1:32000"]

Useful checks:

kubectl describe pod -n <namespace> <pod>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
./bin/mcp-runtime cluster doctor

If events say http: server gave HTTP response to HTTPS client, the node tried HTTPS against the plain HTTP dev registry. Recreate the Kind cluster with the mirror or configure the node runtime for the exact image host.

Host Cannot Push to Cluster DNS Registry

This is expected:

docker push registry.registry.svc.cluster.local:5000/example:dev

The host usually cannot resolve Kubernetes service DNS. Use the CLI helper:

./bin/mcp-runtime auth login --api-url http://localhost:18080

./bin/mcp-runtime registry push \
  --image example:dev \
  --name example

For one-node Kind debug loops, kind load docker-image is also valid when the Deployment image name matches the loaded tag.

Dashboard or API Returns 404

Check the Traefik port-forward and ingress resources:

lsof -nP -iTCP:18080 -sTCP:LISTEN
kubectl get ingress -A
kubectl logs -n traefik deploy/traefik --tail=120

The local dashboard should be reachable at:

http://localhost:18080/

Browser Login Fails but Direct API Works

The HTTP ingress overlay can include the pii-redactor@file Traefik middleware. That middleware is useful for request-path testing on ingest traffic, but it must not be attached to control-plane /api routes because API keys, team IDs, server names, namespaces, and grant/session subjects must stay exact. If local API responses show [redacted], verify that mcp-sentinel-gateway-api does not reference pii-redactor@file. For identity-store debugging, port-forward the API directly:

kubectl port-forward -n mcp-sentinel svc/mcp-sentinel-api 18081:8080

Then use http://localhost:18081 for direct API debugging and stop the port-forward afterward.

Operator Is Reconciling but Routes Do Not Work

Check each layer in order:

kubectl get mcpserver <server-name> -n <namespace> -o yaml
kubectl get deploy,svc,ingress -n <namespace> | rg '<server-name>|NAME'
kubectl describe pod -n <namespace> -l app=<server-name>
kubectl logs -n <namespace> deploy/<server-name> -c mcp-gateway --tail=120
kubectl logs -n traefik deploy/traefik --tail=120

In local Kind, MCPServer.status.phase can stay PartiallyReady even when the Deployment is ready and traffic works, because strict ingress readiness waits for load balancer status. Use the Deployment, Service, Ingress, and actual traffic checks to decide whether local routing works.

Grant or Session Does Not Affect Traffic

Use the platform API path first:

./bin/mcp-runtime auth login --api-url http://localhost:18080
./bin/mcp-runtime access grant list --namespace mcp-servers
./bin/mcp-runtime access session list --namespace mcp-servers
./bin/mcp-runtime server policy inspect <server-name> --namespace mcp-servers

Allow a few seconds after apply; the gateway sidecar reloads rendered policy on a short polling loop. If policy looks correct but calls still fail, check gateway logs and request headers (Mcp-Session-Id, X-MCP-Agent-Session, X-MCP-Human-ID, X-MCP-Agent-ID).

Admin/operator fallback:

kubectl get mcpaccessgrant,mcpagentsession -n <namespace> -o wide
./bin/mcp-runtime server policy inspect <server-name> --namespace <namespace> --use-kube
kubectl get cm -n <namespace> <server-name>-gateway-policy -o yaml