Proposal: Multi-cluster routing via URL path

[BorisTyshkevich]

Proposal: Multi-cluster routing via URL path

Context

Today one altinity-mcp process serves exactly one ClickHouse cluster. The
upstream CH endpoint is fixed at startup via clickhouse.host /
clickhouse.port and every tool invocation forwards to that endpoint. To
serve N clusters we deploy N copies of altinity-mcp with N distinct helm
releases.

This proposal lets one altinity-mcp process serve N ClickHouse clusters by
routing on URL path — the path identifies which cluster the request
targets. All other ClickHouse configuration (port, TLS, mode, view/table
regexes, limits, …) is shared across clusters: only the hostname differs,
and hostnames are derived from the cluster name via a single template
substitution.

The motivating shape is the Altinity Cloud operator naming convention, e.g.
cluster otel → service chi-otel-otel-0-0. A template like
chi-{cluster}-{cluster}-0-0.demo expands at request time.

Non-goals

Explicitly out of scope; revisit in v2:

• Per-cluster config blocks. All clusters share one clickhouse: block.
  If two clusters need different ports, modes, regexes, etc., that's v2.
• Cluster discovery tool (list_clusters etc.). Users will configure
  multiple MCP connectors in Claude manually — one URL per cluster.
• JWE × multi-cluster coexistence. JWE uses path-based token routing
  (/{token}) that conflicts with cluster-name path routing. Config load
  rejects the combination multicluster.path_regex != "" && jwe.enabled.
• Local JWT validation on the MCP request path. Today's injector
  deliberately doesn't validate (the CH-side sidecar is the validator).
  v1.1 doesn't change that; the catalog cache key is the SHA-256 of the
  raw bearer (see "Cache key" below), explicitly not an identity assertion.
• Helm chart restructuring. Only the host: field's semantics expand to
  allow a {cluster} placeholder. Existing values files keep working
  unchanged.
• Per-cluster OpenAPI. OpenAPIHandler / ServeOpenAPISchema read the
  wrapper server's global s.dynamicTools map, populated by a once-gated
  EnsureDynamicTools. In multi-cluster mode that global would be filled
  by whichever cluster's user arrived first, and every subsequent OpenAPI
  request — across all clusters and users — would see the same frozen
  catalog. CHConfigFromContext would route the underlying query
  correctly but the schema itself would leak across tenants. Refactoring
  OpenAPI onto the per-request catalog cache is meaningful work that
  isn't on the critical path for the multi-cluster motivation. v1
  disables OpenAPI in multi-cluster mode at config load; v2 lands the
  refactor.

Design overview

One process, one HTTP server, one global ClickHouseConfig, one new config
directive (multicluster.path_regex) that flips the process into
multi-cluster mode.

In multi-cluster mode, an HTTP middleware:

1. Matches the URL against the cluster regex, extracts the cluster name.
2. Validates the cluster name against the DNS-label regex and optional
   allowlist.
3. Expands {cluster} in clickhouse.host to produce reqCfg, injects
   (cluster, reqCfg) into context, and chains to the existing
   authInjector and serverInjector so the bearer and wrapper server
   land on context too.

Then the SDK's getServer(*http.Request) callback fires per request and:

4. Reads the bearer from context (set by authInjector). If empty,
   returns a static-tools-only server — multi-cluster mode requires OAuth
   so authInjector will normally have already 401'd, but the guard
   prevents an unauthenticated request from ever colliding on the
   empty-bearer cache slot.
5. Derives cacheKey = sha256(bearer) and best-effort exp via
   unverified JWT parse.
6. Calls cache.GetOrDiscover(...) keyed on (cacheKey, cluster); on
   miss, runs full discovery under singleflight.
7. Constructs a fresh *mcp.Server with this user's static + dynamic
   tools, resources, and prompts registered, and returns it.

The SDK then owns everything else — transport guardrails, initialize,
ping, tools/list, tools/call, framing. No custom JSON-RPC dispatcher.

Because each URL path scopes a request to exactly one cluster, the MCP
server appears to clients as N independent MCP servers sharing one process.
Each endpoint has its own tools/list containing the static tools
(execute_query, write_query) plus the views/tables visible to that user
on that cluster.

URL layout

Multi-cluster mode introduces two coupled config fields:

• mount_prefix (default /mcp/) — the literal path prefix where the
  cluster middleware mounts on the outer mux. Must be a literal string, not
  a regex.
• path_regex (default ^/mcp/(?P<cluster>[^/]+)(?P<rest>/.*)?$) — applied
  to requests reaching the mount. Must contain a cluster named group;
  may contain an optional rest named group that captures the suffix used
  to dispatch MCP transport vs OpenAPI within the cluster subtree.

The two are not derived from each other (deriving a mount prefix from an
arbitrary regex is fragile). Config-load validates that path_regex
matches paths that start with mount_prefix.

Pod-internal: the pod sees /mcp/{cluster} for the MCP transport.
OpenAPI is disabled in multi-cluster mode (see non-goals); there is no
per-cluster OpenAPI surface in v1.

Ingress: the external URL is https://mcp.host/mcp/{cluster} with
passthrough. Operators who want a different layout override both fields
consistently.

Path separation between system endpoints and cluster names is enforced
twice — once by mount_prefix (system endpoints live outside it), and
once by the cluster-name validator's leading-dot exclusion (see "Validation
at config load" below). Belt and suspenders: even under an aggressive
mount_prefix: /, .health, .well-known, .livez all fail the
DNS-label regex's first-character class.

Default mux layout (multi-cluster mode):

1. /health, /livez                           (operational, exact-match)
2. /jwe-token-generator                      (registered only when JWE enabled;
                                              incompatible with MC at config-load)
3. /.well-known/...                          (OAuth + MCP discovery, RFC-mandated path)
4. /oauth/...                                (OAuth endpoints; advertised in AS metadata,
                                              cannot be moved)
5. /mcp/{cluster}                            (MCP transport per cluster)

OpenAPI routes (/openapi/...) are not registered in multi-cluster mode —
config load rejects the combination. Routes 1–4 register with
exact-or-prefix patterns on the outer mux; the cluster matcher mounts on
the /mcp/ subtree. http.ServeMux longest-prefix semantics already place
exact-match routes ahead of subtree handlers, so the default layout has no
overlap. Operators choosing a non-default path_regex get the same
protection via the cluster-name regex.

Config schema (additive only)

# Existing ClickHouseConfig — unchanged shape; `host` gains optional {cluster}.
clickhouse:
  host: "chi-{cluster}-{cluster}-0-0.demo"   # {cluster} → URL-derived name
  port: 8443
  protocol: https
  view_regexp: "^v_.*"
  table_regexp: "^t_.*"
  # … all existing fields unchanged

# New top-level section. Presence of path_regex enables multi-cluster mode.
multicluster:
  # Literal mount point on the outer mux. The cluster middleware handles
  # every request under this prefix.
  mount_prefix: "/mcp/"
  # Regex applied to incoming paths reaching the mount. MUST contain a
  # "cluster" named group. v1 matches only the MCP transport path (no
  # trailing suffix); per-cluster OpenAPI is deferred to v2.
  path_regex: "^/mcp/(?P<cluster>[^/]+)/?$"
  # Cluster-name validation. Default is a strict RFC 1123 DNS label.
  cluster_name_regex: "^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$"
  # Optional explicit allowlist; if non-empty, only listed names are accepted.
  cluster_allowlist: []                          # e.g. ["otel", "antalya"]
  catalog_cache_max: 10000                       # hard cap on cache entries
  catalog_ttl_fallback: 15m                      # used when JWT exp is unknown

Path dispatch within the cluster subtree:

match path_regex → cluster
  on match → SDK StreamableHTTPHandler with reqCfg injected on ctx
  else     → 404

Validation at config load:

• PathRegex non-empty → compiles, contains named group cluster.
• MountPrefix non-empty → starts with /, ends with /, contains no
  regex metachars. Defaults to /mcp/ if PathRegex is set but
  MountPrefix is empty.
• PathRegex must accept at least one path of the form
  MountPrefix + "<dns-label>"; checked by matching a synthetic sample at
  startup. Catches the "operator changed one but not the other" footgun.
• PathRegex non-empty AND Server.JWE.Enabled → startup error. (JWE owns
  /jwe-token-generator and the /{token}/... path layout, both
  incompatible with cluster-name path routing.)
• PathRegex non-empty AND Server.OAuth.Enabled == false → startup
  error. Multi-cluster mode requires per-request credentials; without
  OAuth there's no per-user identity for cache keying and the shared
  cfg.ClickHouse cannot meaningfully apply across a templated host.
• PathRegex non-empty AND Server.OpenAPI.Enabled → startup error.
  See the OpenAPI non-goal above; v1 refuses to start with both on.
• ClusterNameRegex compiles (default applied if empty). The default —
  ^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$ — already forbids a leading dot,
  which is the load-bearing property: cluster names cannot collide with
  /.well-known/*, /.health, or any other dot-prefixed system path
  regardless of the mount.
• CatalogCacheMax defaults to 10000, min 100.
• CatalogTTLFallback defaults to 15m, range [1m, 24h].
• clickhouse.host containing {cluster} in single-cluster mode is logged
  as a config warning (works literally, almost certainly a misconfiguration).

Request lifecycle (multi-cluster mode)

HTTP request → outer mux
  │
  ├─ /health, /livez, /oauth/*, /.well-known/*   → operational handlers
  └─ /mcp/{cluster}[...] (under mount_prefix)    → cluster middleware
        │
        1. Apply multicluster.path_regex; extract cluster.
        │    no match → 404
        │
        2. Validate cluster name against cluster_name_regex + allowlist.
        │    fail → 404 "unknown cluster"
        │
        3. Build per-request ClickHouseConfig and inject (cluster, reqCfg)
        │   into ctx:
        │     reqCfg := cfg.ClickHouse
        │     reqCfg.Host = strings.ReplaceAll(... "{cluster}" ... cluster)
        │
        4. Existing auth injector runs (extracts bearer/JWE, no validation).
        │   Multi-cluster mode requires OAuth, so a missing bearer here is
        │   normally rejected as 401 by authInjector before reaching the SDK.
        │
        5. Existing serverInjector runs (puts *ClickHouseJWEServer on ctx).
        │   Preserves the CHJWEServerKey contract — static and dynamic
        │   tool handlers read it via GetClickHouseJWEServerFromContext.
        │
        6. Hand off to SDK StreamableHTTPHandler.
        │
        SDK path: SDK calls getServer(r) → buildServerForRequest(r):
              read bearer from ctx (set by authInjector in step 4)
              if bearer == "" → return static-only server  (defensive guard)
              compute cacheKey = sha256(bearer); exp = best-effort JWT exp
              load-or-discover catalog from cache (key = cacheKey \x00 cluster)
              construct fresh *mcp.Server with HasTools/HasResources/HasPrompts
              register static tools, resources, prompts
              register dynamic tools from the catalog
              return server
        SDK then owns initialize/tools/list/tools/call/ping/...

Cache key (token-bound)

The MCP auth injector intentionally doesn't validate inbound bearers — the
CH-side ch-jwt-verify sidecar is the validator. So we cannot derive a
trust-grade identity at the MCP request boundary.

v1.1 keys the catalog cache on the raw bearer itself, not on claims:

func cacheKey(bearer string) string {
    return "tok:" + sha256_hex(bearer)
}

// Best-effort exp extraction for TTL only. Not authoritative.
func bearerExp(bearer string) (time.Time, bool) {
    if claims, ok := parseJWTUnverified(bearer); ok {
        if e, ok := claims["exp"].(float64); ok && e > 0 {
            return time.Unix(int64(e), 0), true
        }
    }
    return time.Time{}, false
}

Why token-bound, not claim-bound:

• Claim-based hashing (sha256(iss\x00sub\x00aud)) lets an attacker who
  forges a JWT with another user's iss/sub/aud hit that user's cached
  catalog on tools/list, disclosing tool names and descriptions — which
  in this codebase are derived from view/table names and may carry sensitive
  schema metadata. tools/call would fail later at ClickHouse, but the
  catalog has already leaked.
• Token-based hashing eliminates the attack: only the legitimate
  token-holder can produce a bearer that hashes to a given cache key. An
  attacker presenting a different token gets a different key and runs full
  discovery against ClickHouse, which then rejects them.
• Cost: rotation invalidates the cache (one extra discovery on first call
  with a new token). Acceptable.

exp is still extracted via unverified parse for TTL calculation only —
this is not a security boundary; the worst that can happen is a too-short
or too-long TTL (clamped by catalog_ttl_fallback).

The cache key is safe to log (opaque hash of an opaque secret).

Catalog cache

tools/call cannot afford to re-run discovery on every invocation — for
write-enabled setups that costs N+1 ClickHouse round-trips. Discovery on
tools/list is acceptable; tools/list is rare per session and clients
expect it slow.

type catalogEntry struct {
    Tools     map[string]DynamicToolMeta
    ExpiresAt time.Time
}

type catalogCache struct {
    mu      sync.RWMutex
    entries map[string]catalogEntry  // key: cacheKey(bearer) + "\x00" + cluster
    max     int
    sf      singleflight.Group       // collapses concurrent discoveries
}

Discipline:

• Load-on-build. buildServerForRequest calls cache.GetOrDiscover(...)
  which returns the cached catalog on hit, or runs DiscoverTools(...) on
  miss under singleflight.Do(cacheKey + "\x00" + cluster, …). The first
  concurrent request triggers discovery; the rest wait and share the result.
• TTL. ExpiresAt = min(jwtExp, now + catalog_ttl_fallback). If JWT
  exp is unknown (opaque token or unparseable JWT), ExpiresAt = now + catalog_ttl_fallback.
• No sliding TTL, no LRU. Hard expiry only. A janitor goroutine sweeps
  expired entries once per minute.
• Hard cap. When inserting would exceed max, sweep first; if still
  full, drop the insert and log at warn. Bounds worst-case memory at
  O(max × avg_tools_per_user).
• Per-pod, in-memory. No Redis. Pod restart loses the cache; the next
  request rebuilds it.

tools/list is not specially handled — it reads from the per-request
*mcp.Server which was built from the cache. If the operator wants
tools/list to always rediscover, they can lower catalog_ttl_fallback;
that's the tuning knob.

What this is not: a session abstraction. There's no Session type, no
per-connection state. The cache is a TTL-keyed memo of discovery output.

Why per-request *mcp.Server

The SDK 1.6.1's NewStreamableHTTPHandler accepts
getServer func(*http.Request) *Server — it's invoked per incoming request.
Server.AddTool mutates the server's tool map, and the server's
tools/list/tools/call handlers read that map. So a fresh server per
request, populated with that user's catalog, gives us per-(user, cluster)
tools using only documented SDK APIs.

Cost per request:

• One mcp.NewServer with capability flags matching today's server:
  HasTools: true, HasResources: true, HasPrompts: true. Resources and
  prompts are part of the existing server's advertised surface (see
  pkg/server/server.go:54–82) and must remain in multi-cluster mode —
  scoping them out would be a behavior change to existing clients.
• N AddTool(...) calls plus the existing resource/prompt registrations
  (RegisterResources, RegisterPrompts). These are config-static, so
  the same call set runs every request.
• Static tools: 1–2 AddTool calls (execute_query, optionally
  write_query).

The cache reuses the catalog itself across requests; only the server-and-tool
wiring is rebuilt. For typical catalogs (≤50 tools) this is microseconds.

Discovery timing (intentional)

getServer runs on every request, including initialize, ping, and
notifications — not just tools/list and tools/call. With Stateless: true, there is no other request hook. This means:

• On a cold cache, the very first request from a new (token, cluster) pair
  triggers full discovery, regardless of which JSON-RPC method it carries.
• Subsequent requests in the same TTL window hit the cache and skip CH.

This is intentional, not a bug. A typical client opens a connection with
initialize and then quickly calls tools/list — both would have needed
discovery anyway, and doing it on initialize means tools/list is
already warm. The alternative (peek the body to skip discovery on
non-tool methods) would reintroduce body-sniffing complexity for marginal
benefit. If DiscoverTools fails (e.g. CH unreachable, missing creds),
buildServerForRequest falls back to a static-tools-only server so
initialize still succeeds and the client can surface the error via the
first tool call.

Inherited from the SDK for free

• DNS-rebinding Host check (streamable.go:258–265).
• Origin / CSRF check via CrossOriginProtection (streamable.go:267–273).
• Content-Type enforcement (streamable.go:276–278).
• Accept negotiation (streamable.go:283–293).
• HTTP method allowlist with Allow headers (streamable.go:334–357).
• MCP-Protocol-Version validation (streamable.go:391–397).
• Session-ID handling and user-mismatch detection (streamable.go:295–317).
• Full JSON-RPC framing for tool calls, including batched requests.

None of this needs to be re-implemented. The previous v1 design's rpcmux
package is dropped.

In-RAM session structure (v1.1, final)

There is no per-session structure. Stateless: true on the transport stays.
The "session" concept is:

1. Request context carries cluster (string) and reqCfg
   (ClickHouseConfig) — alongside the existing CHJWEServerKey and
   JWE/OAuth token keys. Lives one HTTP request.
2. Catalog cache holds discovery output keyed by (cacheKey, cluster)
   with hard expiry. A TTL-keyed memo of tools/list-equivalent output.
   The cache key is computed inside getServer from the bearer that
   authInjector already put on context; it is not a context value
   of its own.

No Session type, no per-connection state, no cross-pod replication.

Code-level changes

pkg/config/config.go

• New MulticlusterConfig:

  type MulticlusterConfig struct {
      MountPrefix         string        `yaml:"mount_prefix"`          // default /mcp/
      PathRegex           string        `yaml:"path_regex"`
      ClusterNameRegex    string        `yaml:"cluster_name_regex"`    // default RFC 1123
      ClusterAllowlist    []string      `yaml:"cluster_allowlist"`
      CatalogCacheMax     int           `yaml:"catalog_cache_max"`     // default 10000
      CatalogTTLFallback  time.Duration `yaml:"catalog_ttl_fallback"`  // default 15m
  }

• Add Multicluster MulticlusterConfig to Config.
• Validation enforces all the rules in "Validation at config load" above,
  including the synthetic-sample match between MountPrefix and PathRegex.

Package placement

The cluster router and catalog cache live inside pkg/server, not in a
new subpackage. Reason: dynamicToolMeta and the dynamic-tool registration
helpers are unexported (server_dynamic_tools.go:28 and the
s.registerDynamicTools method), and a subpackage cannot reference them.
Exporting them just to allow a subpackage to consume them would widen the
public API for no other caller. Keeping the new code in the same package
avoids that.

New files under pkg/server:

• multicluster_router.go — MulticlusterRouter middleware: applies
  path_regex, validates cluster name, builds reqCfg, injects context.
  Constructor: NewMulticlusterRouter(cfg config.MulticlusterConfig, ch config.ClickHouseConfig) (*MulticlusterRouter, error). Exposes
  (r *MulticlusterRouter) Match(req) (cluster, rest string, ok bool).
• catalog_cache.go — CatalogCache with constructor
  NewCatalogCache(cfg config.MulticlusterConfig) *CatalogCache and method:

  func (c *CatalogCache) GetOrDiscover(
      ctx context.Context,
      cacheKey, cluster string,
      reqCfg config.ClickHouseConfig,
      factory ClientFactory,
      rules []config.DynamicToolRule,
      readOnly bool,
      bearerExp time.Time, // zero → use fallback TTL
  ) (map[string]dynamicToolMeta, error)

  This is the single canonical signature; all other references in this doc
  conform to it.
• multicluster_identity.go — CacheKey(bearer string) string,
  BearerExp(bearer string) (time.Time, bool), and the context helpers
  ClusterFromContext, CHConfigFromContext. No CacheKeyFromContext
  — cacheKey is computed inside getServer from the bearer already on
  context.

Roughly 400 LOC including tests.

pkg/server/server_dynamic_tools.go

The current discovery path reaches credentials via
s.GetClickHouseClientFromCtx(ctx), which extracts JWE / OAuth tokens and
claims from context and merges them with s.Config.ClickHouse as the
host source. In multi-cluster mode the host source must be reqCfg, not
the global — otherwise discovery and tool calls silently route to the
wrong backend.

Discovery takes the resolved config by value and a credential-only
factory:

type ClientFactory func(
    ctx context.Context,
    chCfg config.ClickHouseConfig,   // authoritative — host already expanded
) (*clickhouse.Client, error)

func DiscoverTools(
    ctx context.Context,
    chCfg config.ClickHouseConfig,
    factory ClientFactory,
    rules []config.DynamicToolRule,
    readOnly bool,
) (map[string]dynamicToolMeta, error)

factory must respect chCfg.Host as authoritative; the factory pkg/server
provides reads JWE/OAuth tokens and claims from ctx, then constructs a
*clickhouse.Client against the passed chCfg without consulting the
global. The existing GetClickHouseClientWithOAuth already accepts an
explicit chCfg-equivalent path; we add a tightened variant that refuses
to fall back to s.Config.ClickHouse and route through it from both modes:

• Single-cluster mode keeps calling the factory with
  s.Config.ClickHouse. Behavior unchanged.
• Multi-cluster mode calls it with reqCfg. The factory never sees the
  global, so host expansion is the only authoritative source.

Other changes:

• EnsureDynamicTools becomes a thin caller for single-cluster mode: calls
  DiscoverTools(s.Config.ClickHouse, s.clientFactory, ...) once, then
  s.registerDynamicTools(...) as today.
• Handler factories (makeDynamicToolHandler,
  makeDynamicWriteToolHandler) read CHConfigFromContext(ctx).
  CHConfigFromContext falls back to the global config when the context
  key is absent (single-cluster mode); in multi-cluster mode the router
  middleware always sets the key, so the global is never consulted.

pkg/server/server.go

• New context keys: ClusterNameKey, RequestCHConfigKey. These
  coexist with the existing CHJWEServerKey, JWETokenKey,
  JWEClaimsKey, OAuthTokenKey, OAuthClaimsKey — none of those are
  changed. The multi-cluster middleware chain is router → authInjector
  → serverInjector → sdkHandler: router sets cluster/reqCfg first,
  auth puts the bearer on ctx, serverInjector puts the wrapper server on
  ctx so handlers can find it via GetClickHouseJWEServerFromContext.
• CHConfigFromContext(ctx) config.ClickHouseConfig — returns
  RequestCHConfigKey if set, otherwise the server's global
  s.Config.ClickHouse (single-cluster path).
• No CacheKeyFromContext — cacheKey is derived inside the getServer
  closure from the bearer already on ctx.
• GetClickHouseClientFromCtx switches to reading CHConfigFromContext
  for host/port/TLS, while still extracting JWE/OAuth tokens from ctx as
  today. The internal helper GetClickHouseClientWithOAuth gains an
  explicit chCfg parameter so it never reaches for the global.

Static-tool registration adapter. AltinityMCPServer.AddTool takes our
local ToolHandlerFunc; *mcp.Server.AddTool takes mcp.ToolHandler. The
two function signatures are identical but they are distinct named types so
Go requires a conversion. Add:

// sdkServerAdapter exposes a plain *mcp.Server as an AltinityMCPServer so
// it can be passed to registerStaticTool / registerDynamicTools without
// pulling in the wrapper-server's other state.
type sdkServerAdapter struct{ srv *mcp.Server }

func (a sdkServerAdapter) AddTool(tool *mcp.Tool, h ToolHandlerFunc) {
    a.srv.AddTool(tool, mcp.ToolHandler(h))
}
// AddResource / AddResourceTemplate / AddPrompt mirrored similarly.

// RegisterToolsOnSDKServer registers static + dynamic tools onto a fresh
// *mcp.Server. Used by the multi-cluster getServer closure.
func RegisterToolsOnSDKServer(srv *mcp.Server, cfg *config.Config,
                              dynamic map[string]dynamicToolMeta) {
    adapter := sdkServerAdapter{srv: srv}
    registerStaticToolsOn(adapter, cfg)
    registerDynamicToolsOn(adapter, dynamic)  // pure helper, not the method
}

registerDynamicToolsOn is a new pure helper that takes the catalog map
and an AltinityMCPServer; the existing
(*ClickHouseJWEServer).registerDynamicTools becomes a wrapper that calls
it (so single-cluster behavior is unchanged).

cmd/altinity-mcp/main.go

Three changes around the existing mcp.NewStreamableHTTPHandler setup.

1. Mode select. If cfg.Multicluster.PathRegex == "" → unchanged path.
   The global *mcp.Server is pre-populated with static + dynamic tools
   as today, and getServer returns it.

2. Multi-cluster path. Replace the getServer closure:

   router, err := server.NewMulticlusterRouter(cfg.Multicluster, cfg.ClickHouse)
   if err != nil { return err }
   cache := server.NewCatalogCache(cfg.Multicluster)
   
   // Factory closes over the JWE wrapper for credential extraction but
   // takes chCfg as authoritative (no fallback to s.Config.ClickHouse).
   factory := func(ctx context.Context, chCfg config.ClickHouseConfig) (*clickhouse.Client, error) {
       jwe   := a.mcpServer.ExtractTokenFromCtx(ctx)
       oauth := a.mcpServer.ExtractOAuthTokenFromCtx(ctx)
       claims := a.mcpServer.GetOAuthClaimsFromCtx(ctx)
       return a.mcpServer.GetClickHouseClientWithOAuthForConfig(ctx, chCfg, jwe, oauth, claims)
   }
   
   newSDKServer := func(tools map[string]dynamicToolMeta) *mcp.Server {
       srv := mcp.NewServer(&mcp.Implementation{Name: "altinity-mcp"},
           &mcp.ServerOptions{HasTools: true, HasResources: true, HasPrompts: true})
       server.RegisterToolsOnSDKServer(srv, &cfg, tools)
       server.RegisterResourcesOnSDKServer(srv)
       server.RegisterPromptsOnSDKServer(srv)
       return srv
   }
   
   getServer := func(r *http.Request) *mcp.Server {
       cluster, ok := server.ClusterFromContext(r.Context())
       if !ok { return nil } // SDK returns 400
   
       reqCfg := server.CHConfigFromContext(r.Context())
   
       // authInjector has already populated the bearer on ctx (or 401'd).
       // The empty-bearer guard is defensive: returning a static-only
       // server here prevents any unauthenticated request from colliding
       // on a single empty-bearer cache slot, even under a misconfigured
       // middleware chain.
       bearer := a.mcpServer.ExtractOAuthTokenFromCtx(r.Context())
       if bearer == "" {
           return newSDKServer(nil)  // static tools / resources / prompts only
       }
   
       key    := server.CacheKey(bearer)
       exp, _ := server.BearerExp(bearer)
   
       tools, err := cache.GetOrDiscover(r.Context(), key, cluster, reqCfg,
           factory, cfg.Server.DynamicTools, cfg.ClickHouse.ReadOnly, exp)
       if err != nil {
           log.Warn().Err(err).Msg("multicluster: discovery failed; static-only")
           tools = nil
       }
       return newSDKServer(tools)
   }
   
   sdkHandler := mcp.NewStreamableHTTPHandler(getServer,
       &mcp.StreamableHTTPOptions{Stateless: true})

3. Mux registration. Outer mux registers /health, /livez, /oauth/*,
   /.well-known/*, then mounts the multi-cluster handler chain on
   cfg.Multicluster.MountPrefix. OpenAPI routes are not registered in
   multi-cluster mode (rejected at config load).

   mux.Handle(cfg.Multicluster.MountPrefix,
       corsHandler(stripTrailingSlash(
           router.Middleware(                    // steps 1–3: cluster, reqCfg
               authInjector(                     // step 4: bearer → ctx (or 401)
                   serverInjector(sdkHandler))))) // step 5: wrapper → ctx; SDK
   )
   

   Outer → inner is execution order (outermost middleware runs first).
   router.Middleware does only steps 1–3 — extract cluster, validate
   name, build reqCfg, inject (cluster, reqCfg) into ctx. cacheKey is
   not computed here. serverInjector must remain in the chain — static
   and dynamic tool handlers all call GetClickHouseJWEServerFromContext
   and bail with "can't get JWEServer from context" if it's absent.

helm/altinity-mcp/values.yaml

• Document the {cluster} placeholder semantic on clickhouse.host.
• Add optional multicluster: block with the fields above.
• Existing deployments need no values changes.
• New values_examples/mcp-multicluster.yaml showing the
  chi-{cluster}-{cluster}-0-0.demo shape.

Test plan

Unit tests

• pkg/config: regex validation (missing cluster group rejected, malformed
  regex rejected); MC + JWE combination rejected; defaults applied.
• pkg/server (multicluster files):
  • MulticlusterRouter.Match: positive/negative cases; cluster-name
    regex rejects evil.example, .., IPv4 literals, overlong names,
    leading-dot names (.health, .well-known); allowlist enforced.
  • CatalogCache.GetOrDiscover: hit, miss, TTL sweep, hard-cap behavior,
    concurrent miss collapsed by singleflight (test with a counted fake
    discovery function), fallback TTL when JWT exp absent.
  • CacheKey: stable on identical bearer; differs across rotated tokens;
    two different bearers with identical iss/sub/aud claims hash to
    different keys (regression test for the v1.0 attack); empty bearer
    is never passed to CacheKey from getServer (covered by the
    empty-bearer guard).
  • BearerExp: extracts exp from JWT; returns ok=false for opaque or
    malformed bearer.
  • getServer empty-bearer guard: a request with no Authorization
    header that somehow reaches getServer returns a static-only server
    (no cache.GetOrDiscover call, no CH round-trip).
• pkg/server/server_dynamic_tools: DiscoverTools(ctx, chCfg, factory, rules, readOnly) calls the factory once per discovery; handler factories
  read CHConfigFromContext.
• pkg/server (adapter): sdkServerAdapter.AddTool correctly converts
  ToolHandlerFunc → mcp.ToolHandler; tool invocation through the
  adapter produces identical results to direct registration.
• pkg/server: host expansion ({cluster} substitution; literal host with
  warning).
• cmd/altinity-mcp: mux ordering — /health, /livez, /oauth/*,
  /.well-known/* are not interpreted as cluster names.

Integration / e2e (post-deploy)

End-to-end against the otel demo cluster:

1. Deploy with:

   clickhouse:
     host: "chi-{cluster}-{cluster}-0-0.demo"
     port: 8443
     protocol: https
     mode: gating
   multicluster:
     mount_prefix: "/mcp/"
     path_regex: "^/mcp/(?P<cluster>[^/]+)/?$"
     cluster_allowlist: ["otel", "antalya"]
   openapi:
     enabled: false  # required in multi-cluster mode for v1

2. Configure two MCP connectors in Claude:
   • https://mcp.../mcp/github
   • https://mcp.../mcp/antalya
3. With each connector:
   • tools/list returns static tools + cluster-specific views.
   • execute_query succeeds against the correct backend.
   • /mcp/bogus → 404 "unknown cluster" (allowlist).
   • /mcp/evil.example → 404 (DNS-label regex).
   • /mcp/otel/openapi/list_tables → 404 (OpenAPI disabled in MC mode).
4. Single-cluster regression: a deployment without multicluster.path_regex
   behaves identically to today (including resources, prompts, OpenAPI).
5. Config-load rejection: a config with multicluster.path_regex set AND
   openapi.enabled: true refuses to start.
6. JWE regression: JWE deployments unaffected (MC + JWE rejected at startup).
7. Concurrency: open both connectors simultaneously with a cold cache;
   verify only one discovery per (cacheKey, cluster) hits CH (sidecar log
   count, or a custom metric exposed by the cache).

Use the test-mcp-connector skill for the e2e step.

Health and liveness endpoints

/livez and /health stay on the outer mux, outside mount_prefix. The
cluster middleware never sees them, and the leading-dot rule in the
cluster-name regex prevents a misconfigured mount from ever interpreting
health or livez as a cluster name.

Neither endpoint changes shape in multi-cluster mode:

• /livez is a pure process check that never touches ClickHouse. It
  reports {status:"alive"} for any running pod regardless of how many
  clusters are configured.
• /health already short-circuits its ClickHouse-ping branch when JWE or
  OAuth is enabled (the existing credentialsArePerRequest gate at
  main.go:689). Multi-cluster mode requires OAuth (enforced at config
  load), so the gate is always true; /health returns 200 with
  auth: "per_request_credentials" — identical to today's OAuth
  deployments.

This avoids the per-cluster-readiness question entirely: there is no
single answer to "is ClickHouse reachable" across N clusters with N
per-user credential sets, and probing each cluster on every health check
would amplify CH load proportional to (clusters × replicas × probe rate).
A future v2 may add an explicit multi-cluster readiness aggregator (e.g.
/readyz?cluster=otel) if a deployment needs it; v1 doesn't.

Backward compatibility

Pure superset of today's behavior:

• multicluster.path_regex unset → behavior unchanged. Single-cluster
  deployments need zero config changes.
• clickhouse.host without {cluster} → behavior unchanged.
• All existing tools, handlers, and CH client construction logic keep
  working; they now read reqCfg from context in multi-cluster mode and
  from the server's global config otherwise. serverInjector and the
  CHJWEServerKey contract are preserved in both modes.
• Resources and prompts continue to be advertised and registered in
  multi-cluster mode (per-request server uses
  HasTools/HasResources/HasPrompts matching today's defaults).
• Helm chart: only host: semantics expand.

Future work (v2 and beyond)

1. list_clusters tool scoped to clusters reachable for the calling user.
2. Per-cluster overrides (cluster_overrides: { name: { … } }) or full
   clusters: map with default_cluster.
3. Local JWT validation on the request path. Today's "validate at CH
   only" model has a soft trust boundary; landing this would let the cache
   key migrate from raw-bearer hash to verified-claim hash (stable across
   token rotation) without re-opening the v1.0 forgery attack. Needs
   introspection-or-JWKS plumbing on the inbound side.
4. Per-cluster OpenAPI. Requires refactoring OpenAPIHandler /
   ServeOpenAPISchema off the wrapper's global s.dynamicTools map and
   onto the per-(cacheKey, cluster) catalog cache. Probably an additive
   OpenAPIHandlerForCatalog(cat) entry point invoked by a
   /mcp/{cluster}/openapi[/...] route added to path_regex (the rest
   group from earlier drafts). Until then, OpenAPI is rejected at config
   load alongside multi-cluster.
5. JWE × multi-cluster coexistence via a path layout that carries both
   token and cluster (e.g. /{token}/{cluster}).
6. Negative caching of 401/403 per (identity, cluster).
7. Sliding TTL / background refresh for catalogs.
8. tools/list_changed push once the spec settles on its post-PR-2322
   server→client model.

[BorisTyshkevich]

Caveats

1. request-scoped ClickHouse config is not wired into the current client path. Today GetClickHouseClientFromCtx delegates to GetClickHouseClientWithOAuth, and the non-JWE path uses s.Config.ClickHouse as the base config. The redesign says middleware injects a per-request reqCfg, but implementation must add an explicit ClickHouseConfigFromContext path and make discovery, static tools, dynamic tools, and resources all use it. Otherwise the router can extract /mcp/{cluster} perfectly and still connect to the global/static host.

2. the current dynamic-tool lifecycle is process-global and mutating. ClickHouseJWEServer owns one dynamicTools map and a dynamicToolsInit boolean. EnsureDynamicTools is explicitly once-per-server, and registers discovered tools into that server. In multi-cluster mode, the old dynamicToolsInjector must be bypassed or replaced; otherwise the first authenticated request can still poison the global server/catalog model.

3. OAuth resource identity is underspecified for “one connector URL per cluster.” The current OAuth challenge builds a single resource metadata URL from global config or request host, not from /mcp/{cluster}. This may be fine if all clusters intentionally share one OAuth audience/resource and ClickHouse-side validation does cluster authorization. But the proposal should say that explicitly. If the intended resource/audience is per cluster, the design needs cluster-aware protected-resource metadata and token audience handling.

4. unvalidated bearer + per-token discovery creates a DoS surface. The current auth middleware accepts any present OAuth bearer and stores it on context without local validation, relying on ClickHouse-side validation. That is compatible with the existing architecture, but multi-cluster discovery means many unique fake tokens can cause many cache misses and ClickHouse connection attempts. Singleflight helps only for duplicate keys, not unique bogus tokens. Add negative caching, global discovery concurrency limits, and metrics before exposing this broadly.

5. cache TTL needs a hard cap even when JWT exp exists. “Prefer JWT exp” is risky if a token has a long lifetime or permissions change before expiry. Use min(exp-now, maxCatalogTTL) and treat the fallback TTL as only one input. Also define eviction policy for catalog_cache_max; “hard cap 10000” is not enough without LRU/random/drop behavior and memory sizing.

6. config/reload semantics need to be declared. Current HTTP startup captures an MCP server in the SDK handler callback. Reload creates a new MCP server, but does not rebuild the HTTP server/mux. For multicluster, changing mount_prefix, path_regex, OAuth metadata, OpenAPI compatibility, or cache settings probably must require restart unless you deliberately refactor routing reload too.

7. the config shape needs care with duration flags. The current reflected CLI flag generator supports string, bool, int, string slices, and string maps; int64 uses strconv.Atoi. So a YAML value like catalog_ttl_fallback: 15m is fine only if implemented as a parsed string or unflagged duration; do not tag a time.Duration field with default "15m" using the existing flag generator.

8. the proposal text itself has a path/OpenAPI inconsistency. It discusses a default regex with optional rest and sub-mux dispatch, but later says v1 matches only the MCP transport path and OpenAPI is disabled. Pick one. For v1, I would keep it simpler: no rest, no per-cluster OpenAPI, exact /mcp/{cluster} plus optional trailing slash only.

Implementation gate would be:

• Add MulticlusterConfig plus startup validation, including “multi-cluster requires OAuth, rejects JWE and OpenAPI, and requires {cluster} in clickhouse.host unless explicitly overridden.”
• Add request-scoped ClickHouse config context plumbing and tests proving /mcp/a and /mcp/b hit different hosts.
• Replace global dynamic discovery with a pure catalog-discovery function returning metadata, then build a fresh SDK server from cached metadata.
• Disable the old dynamicToolsInjector in multicluster mode.
• Add cache TTL cap, eviction policy, negative cache, and discovery concurrency limit.
• Decide whether OAuth resource identity is shared across clusters or cluster-specific.
• State that multicluster routing config is restart-only, or refactor reload/mux behavior.
• Add e2e tests for tools/list, tools/call, invalid cluster, invalid bearer, cache isolation, token rotation, and config incompatibilities.

[BorisTyshkevich]

Design updates (post-review)

Config shape — simplified

Drop mount_prefix, path_regex, and cluster_name_regex from MulticlusterConfig. Final shape:

multicluster:
  cluster_allowlist: ["otel", "antalya"]  # optional; omit or empty = allow all valid DNS labels
  catalog_cache_max: 10000                # optional; default 10000
  catalog_ttl_fallback: 15m              # optional; default 15m

mount_prefix and path_regex were coupled fields that required cross-validation to avoid a footgun (changing one without the other silently breaks routing). Operators who need a non-default URL layout should use ingress path rewriting — that is the right abstraction boundary.

cluster_name_regex is always RFC 1123: ^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$. This is a constant, not a tuning knob.

cluster_allowlist semantics: empty or omitted = allow all valid DNS-label cluster names. The allowlist is an operational scoping tool, not a security gate — in gating mode, ClickHouse-side auth is the real boundary; an unrecognized cluster name produces a CH connection failure regardless.

URL layout — static invariants (documented, not configured)

/mcp/{cluster}              MCP transport (StreamableHTTPHandler)
/mcp/{cluster}/openapi      OpenAPI schema (GET)
/mcp/{cluster}/openapi/{tool}  Dynamic tool endpoint (POST)
/mcp/{cluster}/openapi/execute_query  Raw SQL endpoint (GET)

These are fixed path conventions, not derived from config. Sub-routing between MCP transport and OpenAPI is dispatched inside the cluster middleware.

OpenAPI — included in v1, not deferred

Per-cluster OpenAPI is included in v1. Rationale: the catalog cache (GetOrDiscover) is the hard part and is already in v1; OpenAPI is a thin consumer (~100 LOC) that reads from the same cache. Deferring produces a v2 that exists only to remove a startup rejection.

OpenAPI design:

• Schema is per-(user, cluster). ServeOpenAPISchema calls cache.GetOrDiscover(bearer, cluster) instead of reading s.dynamicTools. Schema reflects exactly the views/tables visible to that user on that cluster.
• Bearer required. GET /mcp/{cluster}/openapi returns 401 if no valid bearer is present. Same auth requirement as tools/list.
• Not HTTP-cacheable. Schema is user-specific; no Cache-Control or ETag on the response.

Startup rejections (updated)

• multicluster block present + jwe.enabled → startup error (path conflict)
• multicluster block present + oauth.enabled == false → startup error (no per-request identity for cache keying)
• clickhouse.host without {cluster} in multicluster mode → config warning (works literally, almost certainly a misconfiguration)
• multicluster block present + openapi.enabled → startup error removed: OpenAPI is supported in multi-cluster mode as of v1.

Resolves open items from the previous comment thread

• Caveat 8 (path/OpenAPI inconsistency): resolved by the static path layout above. No rest group ambiguity — two fixed route prefixes, explicit dispatch.
• "Per-cluster OpenAPI deferred to v2": promoted to v1 per the above.

[BorisTyshkevich]

v1 readiness — final decisions

Follow-up to the post-review design update above. Four decisions land the remaining items needed for implementation.

1. Negative caching (v1)

The catalog cache memoizes auth-class failures from ClickHouse / the sidecar in addition to successes. This does not require local JWT validation — it caches CH's rejection of an opaque bearer, not a trust-grade identity claim.

• catalogEntry carries outcome ∈ {ok, denied}. denied carries no catalog payload, only the error class for telemetry.
• A denied entry is inserted only when DiscoverTools fails with an auth-class error: HTTP 401/403 from the sidecar/CH, or CH error codes 516 AUTHENTICATION_FAILED, 519 REQUIRED_PASSWORD, 497 ACCESS_DENIED. Transient errors (timeout, 5xx, DNS, connection refused) are never cached — caching them would turn a CH/sidecar blip into a multi-second lockout for affected users.
• Hit path in getServer: a denied entry returns a static-only server immediately, no CH round-trip, no discovery slot consumed.
• New knob: catalog_negative_ttl (default 60s, range [10s, 5m]). Separate from catalog_ttl_fallback because the staleness criteria differ — denials should refresh quickly after upstream remediation, while positive entries piggyback on JWT exp.

Cache key is unchanged: sha256(bearer) + "\x00" + cluster. An attacker with a forged bearer gets their own negative entry, never colliding with a legitimate user's positive entry — preserves the v1.0 forgery mitigation.

2. Cache eviction (v1) — random eviction of negatives at 70%

Single cache, capacity catalog_cache_max (default 10000), holds both ok and denied entries. Eviction is asymmetric: negatives are disposable, positives are not.

Constant negativeEvictionThreshold = 0.7 (hardcoded, not exposed): when total fill ≥ 0.7 × catalog_cache_max and at least one denied entry exists, the insert path evicts one random denied entry before proceeding. Uniform random selection, not LRU — no access-time tracking required.

Insert algorithm (executed under cache mutex):

1. If len(entries) >= 0.7 * max  AND  len(deniedKeys) > 0:
       k := deniedKeys[rand.IntN(len(deniedKeys))]
       delete(entries, k); remove k from deniedKeys
2. If len(entries) >= max:
       Drop the new entry, log at warn ("catalog cache full"),
       increment counter catalog_cache_full_drops_total{outcome=...}.
       Return.
3. Insert the entry.
   If outcome == denied: append key to deniedKeys.

Properties:

• Under sustained attack (unique bogus bearers): once fill crosses 70%, every insert evicts one random denied. Steady state oscillates around 70% fill; attacker entries churn out faster than they accumulate; legitimate ok entries are never displaced. Discovery-load amplification is bounded — each unique attacker token forces at most one CH discovery, singleflight collapses duplicates.
• Under legitimate scale: if 70%+ of the cache is ok entries, the 70% sweep is a no-op (no negatives to evict), fill grows to 100% normally, step 2 starts dropping inserts with a warn log + counter. The operational signal "raise catalog_cache_max" is then unambiguous.

Implementation shape:

type catalogCache struct {
    mu          sync.Mutex
    entries     map[string]catalogEntry
    deniedKeys  []string  // parallel index for O(1) random sampling
    max         int
    sf          singleflight.Group
}

math/rand/v2.IntN is fine — eviction isn't security-sensitive, and the package-level RNG is concurrency-safe and fast. The existing once-per-minute janitor also drops expired denied entries and keeps deniedKeys consistent via swap-remove.

Companion mitigation: global discovery concurrency limit. A semaphore counting in-flight DiscoverTools calls; when saturated, getServer returns a static-only server with a warn log + counter. Independent of the cache — prevents unique-token floods from monopolizing CH connection slots even before the negative cache has a chance to memoize them.

3. OpenAPI — postponed to v2

The post-review promotion of OpenAPI to v1 is reverted. v1 multi-cluster scope is MCP transport only.

Reinstated startup rejection: multicluster.path_regex != "" && openapi.enabled == true → config-load error.

URL surface in multi-cluster mode collapses to:

/health, /livez                                          (operational)
/jwe-token-generator                                     (rejected at config-load if MC enabled)
/.well-known/oauth-protected-resource                    (host-root PRM)
/.well-known/oauth-protected-resource/mcp/{cluster}      (RFC 9728 §3.1 strict)
/mcp/{cluster}/.well-known/oauth-protected-resource      (claude.ai / ChatGPT auto-probe path)
/oauth/...                                               (OAuth endpoints)
/mcp/{cluster}                                           (MCP transport)

Per-cluster OpenAPI remains future work v2 (issue body §"Future work" item 4). Refactoring OpenAPIHandler / ServeOpenAPISchema off the wrapper's global s.dynamicTools map and onto the per-(cacheKey, cluster) catalog cache is real work not justified by v1's motivation.

4. Reload semantics — restart-only for multicluster.*

```markdown

Reload semantics

multicluster.* fields are restart-only. Changing cluster_allowlist,
catalog_cache_max, catalog_ttl_fallback, or catalog_negative_ttl
requires a pod restart to take effect.

Rationale: the outer HTTP mux and the cluster router are constructed once
at startup and captured in the SDK's getServer closure. Hot-swapping these
without rebuilding the listener and the catalog cache would leak old
routes/entries.

The existing config-reload path (server.mcpServer hot-swap) continues to
apply to non-routing fields (clickhouse., dynamic-tool rules, OAuth
identity policy). It explicitly skips multicluster.* and logs a
config-reload warning if any multicluster. key changed without a
restart — surfacing the operator-intent mismatch as a discoverable log
line rather than a silent no-op.

A v2 may add a dedicated routing-reload path; v1 doesn't.
```

Caveat 3 (resource identity) — resolved by decision

v1 uses a single shared OAuth resource/audience across all clusters in one process:

resource = aud = oauth.public_resource_url   (pod-wide, no {cluster} expansion)

Implication: a token issued for this MCP grants access to every cluster on which the calling user is provisioned at ClickHouse. Per-cluster isolation is enforced at ClickHouse, via http_authenticator user grants — not at the token-audience layer. Operators MUST ensure CREATE USER bindings reflect the intended per-cluster boundary. Sidecar audience config is identical across all CH pods.

Future work (v2): per-cluster audiences would require cluster-aware PRM bodies, per-cluster sidecar audience, and cluster-aware resource_metadata= in WWW-Authenticate. Not on v1's critical path.

Per-cluster PRM routes (independent of audience decision)

claude.ai and ChatGPT's connector-add flow probes <connector-url>/.well-known/oauth-protected-resource before any 401 fires, so the WWW-Authenticate: resource_metadata= hint never plays into the initial auto-discovery. PRM must be served at the per-cluster path or users will have to enter the metadata URL manually when adding the connector.

Register three routes on the outer mux in multi-cluster mode:

GET /.well-known/oauth-protected-resource                  → host-root (shared-audience JSON)
GET /.well-known/oauth-protected-resource/mcp/{cluster}    → RFC 9728 §3.1 strict
GET /mcp/{cluster}/.well-known/oauth-protected-resource    → claude.ai/ChatGPT auto-probe

All three return the same JSON body under shared-audience v1 — the duplication is purely for client-discovery compatibility. Per-cluster PRM bodies (different resource per cluster) are deferred to the v2 per-cluster-audience work.

Also: the WWW-Authenticate challenge built by oauthChallengeHeader (cmd/altinity-mcp/oauth_server.go:367+) must point resource_metadata= at the per-cluster PRM URL (not host-root) once ClusterFromContext is set, so the 401-path hint agrees with the auto-probe path.

Status

v1 readiness item	Status
Cache-fill DoS / discovery DoS	Closed — negative cache + 70% random eviction of negatives + global discovery concurrency limit
OAuth resource identity	Closed — shared audience, documented
Config reload semantics	Closed — multicluster.* declared restart-only, mismatch logged on reload
OpenAPI in v1	Closed — reverted to v2; startup rejects MC + OpenAPI

Housekeeping for the implementation phase (not design-level)

• catalog_ttl_fallback and catalog_negative_ttl are time.Duration — verify the reflected CLI flag generator handles them, or mark them yaml-only. The existing generator's strconv.Atoi path will misinterpret values like 15m.
• Short context deadline (~3s) on the DiscoverTools call triggered by the cold-cache initialize request; fall back to static-only on timeout so initialize itself never blocks the client.
• The three s.Config.ClickHouse reachback sites in pkg/server/server_client.go:23,56,211 all need the chCfg-explicit variant so multi-cluster mode can't accidentally fall back to the global host.
• e2e test for the auto-discovery path: curl https://multi-mcp.../mcp/otel/.well-known/oauth-protected-resource must 200 with a valid PRM body before claude.ai's connector-add will succeed without manual URL entry.

[BorisTyshkevich]

Deploying altinity-mcp today requires operators to understand two orthogonal concepts:

1. mode: forward vs mode: gating — encodes whether CH accepts Bearer or Basic auth (i.e. whether the ch-jwt-verify sidecar is running). Operators shouldn't need to know this.
2. broker_upstream: true — enables the OAuth AS flow for non-CIMD IdPs like Google. Only works under gating mode, so you need both fields.

Goal: Replace with a single broker: true flag. CH auth format (Bearer vs Basic) is auto-detected on first request per endpoint and cached. mode/broker_upstream kept for backward compat with deprecation warnings.