{"code":0,"data":{"id":523,"title":"Kustomize includeSelectors 陷阱：多实例 LiteLLM 跨实例路由问题排查","content":"## 背景\n\n我们用 LiteLLM 搭建了一个 LLM Proxy Gateway，通过 Kustomize 的 base + instances 结构在同一个 Kubernetes namespace 里部署多个实例，共享一个 PostgreSQL 做统一的 API key 管理和用量追踪：\n\n| 实例 | 域名 | 上游 | Provider |\n|------|------|------|----------|\n| kelly | llm.goodvision.tech | kellycloudai.com | `openai/*` |\n| yxaiapp | ai.goodvision.tech | yxaiapp.com | `openai/*` |\n| claude | claude.goodvision.tech | Anthropic API | `anthropic/*` |\n\n每个实例用 `model_name: \"*\"` 做通配路由，透明转发任意模型名到各自的上游。Kustomize 通过 `namePrefix` 区分资源名，通过 `labels` 打上 `instance` 标签。\n\n```plantuml\n@startuml\n!theme plain\nskinparam backgroundColor #FEFEFE\nskinparam componentStyle rectangle\n\ncloud \"Clients\" {\n  [JMS Test Client] as client\n}\n\npackage \"K8s Namespace: litellm\" {\n  [Ingress\\nclaude.goodvision.tech] as ing\n  [Service: claude-litellm] as svc\n  [Pod: claude-litellm] as pod_claude\n  [Pod: litellm (kelly)] as pod_kelly\n  [Pod: yxaiapp-litellm] as pod_yxaiapp\n  database \"PostgreSQL\" as db\n}\n\ncloud \"Upstream\" {\n  [Anthropic API] as anthropic\n  [kellycloudai.com] as kelly_up\n  [yxaiapp.com] as yxaiapp_up\n}\n\nclient --> ing\ning --> svc\nsvc --> pod_claude : 期望\nsvc ..> pod_kelly : <color:red>意外!</color>\nsvc ..> pod_yxaiapp : <color:red>意外!</color>\npod_claude --> anthropic\npod_kelly --> kelly_up\npod_yxaiapp --> yxaiapp_up\npod_claude --> db\npod_kelly --> db\npod_yxaiapp --> db\n@enduml\n```\n\n## 问题现象\n\n通过 JMS 测试 `claude.goodvision.tech` 时，Spend Logs 里出现了诡异的交替模式：\n\n```\n08:29:12 PM  openai/claude-opus-4-6     15633 tokens  cost: -\n08:29:03 PM  anthropic/claude-opus-4-6   14333 tokens  cost: $0.076\n08:28:54 PM  openai/claude-opus-4-6      168 tokens    cost: -\n08:28:46 PM  openai/claude-opus-4-6      13498 tokens  cost: -\n08:28:40 PM  anthropic/claude-opus-4-6   12355 tokens  cost: $0.065\n08:28:29 PM  openai/claude-opus-4-6      130 tokens    cost: -\n08:28:21 PM  openai/claude-opus-4-6      9243 tokens   cost: -\n08:28:15 PM  anthropic/claude-opus-4-6   2527 tokens   cost: $0.016\n```\n\nClaude 实例配置的是 `anthropic/*`，但请求一会儿匹配到 `anthropic/claude-opus-4-6`，一会儿匹配到 `openai/claude-opus-4-6`。后端变成了 round-robin，多个后端都被 hit 到。\n\n## 排查过程\n\n### 1. 排除数据库交叉污染\n\n三个实例共享 PostgreSQL，首先怀疑数据库里缓存了其他实例的 model 配置：\n\n```sql\nSELECT * FROM \"LiteLLM_ProxyModelTable\";\n-- (0 rows)\n\nSELECT * FROM \"LiteLLM_Config\";\n-- (0 rows)\n```\n\n数据库里没有额外的 model 配置，排除。\n\n### 2. 检查 Deployment 和 Service\n\n```bash\n$ kubectl -n litellm get deployments -o wide\nNAME              READY   SELECTOR\nclaude-litellm    1/1     app=litellm    # <-- 都一样！\nlitellm           1/1     app=litellm\nyxaiapp-litellm   1/1     app=litellm\n```\n\n三个 Deployment 的 selector 全是 `app=litellm`。\n\n### 3. 关键证据：Endpoints\n\n```bash\n$ kubectl -n litellm get endpoints\nNAME              ENDPOINTS\nclaude-litellm    172.16.0.153:4000,172.16.0.236:4000,172.16.0.238:4000\nlitellm           172.16.0.153:4000,172.16.0.236:4000,172.16.0.238:4000\nyxaiapp-litellm   172.16.0.153:4000,172.16.0.236:4000,172.16.0.238:4000\n```\n\n**每个 Service 都有 3 个 Endpoint！** `claude-litellm` Service 把流量 round-robin 分发到了所有三个 pod。\n\n## 根因分析\n\n问题出在 Kustomize 的 `labels` 配置：\n\n```yaml\n# kustomization.yaml (所有实例)\nlabels:\n  - pairs:\n      instance: claude\n    includeSelectors: false  # <-- 罪魁祸首\n```\n\n`includeSelectors: false` 意味着 `instance` label **只加到资源的 metadata.labels**，不会加到：\n\n- Service 的 `spec.selector`\n- Deployment 的 `spec.selector.matchLabels`\n- Pod template 的 `metadata.labels`\n\n所以实际生成的资源是：\n\n```yaml\n# Service: claude-litellm\nmetadata:\n  labels:\n    instance: claude  # ✅ metadata 有\nspec:\n  selector:\n    app: litellm      # ❌ selector 没有 instance!\n\n# Pod: claude-litellm-xxx\nmetadata:\n  labels:\n    app: litellm      # 只有这个\n    # instance: claude 不在这里\n```\n\nService 用 `app=litellm` 做选择，自然命中了 namespace 里所有带 `app=litellm` 的 pod。\n\n```plantuml\n@startuml\n!theme plain\n\nstate \"includeSelectors: false (Bug)\" as bug {\n  state \"Service selector\" as s1 : app=litellm\n  state \"Pod A labels\" as p1 : app=litellm\n  state \"Pod B labels\" as p2 : app=litellm\n  state \"Pod C labels\" as p3 : app=litellm\n  s1 --> p1 : match ✅\n  s1 --> p2 : match ✅\n  s1 --> p3 : match ✅\n}\n\nstate \"includeSelectors: true (Fix)\" as fix {\n  state \"Service selector\" as s2 : app=litellm\\ninstance=claude\n  state \"Pod A labels\" as p4 : app=litellm\\ninstance=claude\n  state \"Pod B labels\" as p5 : app=litellm\\ninstance=kelly\n  state \"Pod C labels\" as p6 : app=litellm\\ninstance=yxaiapp\n  s2 --> p4 : match ✅\n  s2 -[#red]-> p5 : no match ❌\n  s2 -[#red]-> p6 : no match ❌\n}\n\n@enduml\n```\n\n## 修复\n\n### 1. 修改 kustomization.yaml\n\n所有实例的 `includeSelectors` 改为 `true`：\n\n```yaml\nlabels:\n  - pairs:\n      instance: claude\n    includeSelectors: true  # 加到 selector 和 pod labels\n```\n\n### 2. 重建 Deployment\n\nDeployment 的 `spec.selector.matchLabels` 是 **immutable** 的，不能直接更新，必须先删后建：\n\n```bash\n# 删除旧 deployment\nkubectl -n litellm delete deployment litellm claude-litellm yxaiapp-litellm\n\n# 重新 apply\nkubectl apply -k deploy/instances/kelly\nkubectl apply -k deploy/instances/claude\nkubectl apply -k deploy/instances/yxaiapp\n```\n\n### 3. 验证\n\n```bash\n$ kubectl -n litellm get pods --show-labels\nNAME                          LABELS\nclaude-litellm-xxx            app=litellm,instance=claude\nlitellm-xxx                   app=litellm,instance=kelly\nyxaiapp-litellm-xxx           app=litellm,instance=yxaiapp\n\n$ kubectl -n litellm get endpoints\nNAME              ENDPOINTS\nclaude-litellm    172.16.0.155:4000   # ✅ 只有 1 个\nlitellm           172.16.0.154:4000   # ✅ 只有 1 个\nyxaiapp-litellm   172.16.0.240:4000   # ✅ 只有 1 个\n```\n\n每个 Service 现在只路由到自己的 Pod。\n\n## 教训\n\n### Kustomize labels 的 includeSelectors 默认行为\n\n`includeSelectors: false` 是 Kustomize labels transformer 的默认值。它的设计意图是「只打标签，不影响选择逻辑」，适用于：\n- 给资源打 metadata 标签做分类/查询\n- 不需要影响 Service → Pod 的路由关系\n\n但在**同一 namespace 部署多个同类应用实例**的场景下，**必须用 `includeSelectors: true`**，否则所有 Service 会共享 selector，导致跨实例 round-robin。\n\n### 快速检查方法\n\n如果你怀疑有类似问题，直接看 endpoints 数量：\n\n```bash\nkubectl get endpoints -n <namespace>\n```\n\n每个 Service 的 endpoint 数量应该等于对应 Deployment 的 replicas 数。如果 endpoint 数量异常多，大概率是 selector 太宽泛了。\n\n### 为什么 namePrefix 不够\n\nKustomize 的 `namePrefix` 只改资源名（Deployment、Service、ConfigMap 等），不改 label selector。资源名不同不代表 selector 不同——selector 才是决定 Service 路由的关键。","acl":"public","tags":["kubernetes","kustomize","litellm","debugging","devops"],"category_id":4,"category":{"id":4,"name":"系统可靠性工程","description":"关于SRE实践、工具和技术的文章。","color":"#2B1B43","created_at":"2025-06-19T04:29:04.749+08:00","updated_at":"2025-06-19T04:29:04.749+08:00"},"author":{"id":1,"username":"kiyor","email":"cai@kiyor.com"},"enable_variables":false,"created_at":"2026-02-26T12:47:19.256+08:00","updated_at":"2026-02-26T12:47:19.256+08:00"}}