[{"data":1,"prerenderedAt":402},["ShallowReactive",2],{"article:column:/column/cloudnative/health-and-capacity":3},{"id":4,"title":5,"body":6,"description":391,"extension":392,"meta":393,"navigation":397,"path":398,"seo":399,"stem":400,"__hash__":401},"column/column/cloudnative/health-and-capacity.md","健康检查与弹性为什么会适得其反",{"type":7,"value":8,"toc":371},"minimark",[9,13,16,19,22,26,33,36,39,42,45,57,67,70,74,77,80,83,86,90,93,174,187,190,193,196,199,203,209,212,215,222,225,233,236,239,242,245,248,251,254,257,260,267,270,273,278,281,284,287,290,293,297,300,303,306,309,312,316,319,322,325,328,331,334,367],[10,11,12],"p",{},"读书笔记服务增加了全文索引。新版本启动时，需要加载索引并建立连接，耗时约四十秒。开发者设置了一个每五秒执行的健康检查，连续三次失败就重启容器。",[10,14,15],{},"结果，应用始终没有机会完成启动。",[10,17,18],{},"另一次，数据库短暂变慢。所有 API 实例的健康接口都检查数据库，一旦查询超时就返回失败。平台于是同时重启多个本来还能处理部分请求的实例。重启产生更多连接尝试，数据库压力继续增加。",[10,20,21],{},"自动恢复能否帮忙，取决于它收到的信号是否对应正确的动作。",[23,24,25],"h2",{"id":25},"三种探针对应三种决定",[10,27,28,29],{},"Kubernetes 的探针应该从动作理解。",[30,31,32],"span",{},"1",[10,34,35],{},"Readiness 回答“现在是否适合接收这类流量”。普通 Service 的后端选择会利用就绪状态，让尚未准备好的实例暂时退出常规转发。它通常不需要重启进程。",[10,37,38],{},"Liveness 回答“这个容器是否进入了需要重启才能恢复的状态”。例如主工作线程永久卡死，而且进程内部已没有有效恢复路径。失败达到阈值后，kubelet 会按相应重启规则处理容器。",[10,40,41],{},"Startup 给慢启动的应用留出初始化窗口。在配置 startup probe 的情况下，它成功前，liveness 和 readiness 不会开始正常执行。成功后，运行期的探针才承担各自职责。",[10,43,44],{},"对于索引加载场景，启动探针的失败容忍窗口应覆盖合理的最慢启动时间，并留下余量。这个时间应该来自观察，而不是所有项目都复制三十秒。",[10,46,47,48,52,53,56],{},"探针还需要一个清楚的语义。例如 ",[49,50,51],"code",{},"/livez"," 只检查应用本身是否还能继续推进工作；",[49,54,55],{},"/readyz"," 检查当前是否具备服务关键请求的条件。两个路径可以不同，也可以共用底层检测，但不能因为名字不同，就假设它们有了不同含义。",[10,58,59],{},[60,61,63],"a",{"href":62},"/collections-assets/cloudnative/cn08-probe-decisions.png",[64,65],"img",{"alt":66,"src":62},"startup 保护初始化，通过之前 readiness 和 liveness 暂不执行；正常期 readiness 失败退出常规新流量，liveness 持续失败重启容器。",[10,68,69],{},"探针有不同动作，慢启动与下游暂时故障不应都触发同一种重启。",[23,71,73],{"id":72},"把所有依赖都放进-liveness-的代价","把所有依赖都放进 liveness 的代价",[10,75,76],{},"数据库故障通常不是重启 API 就能修复的。把数据库连通性写进 liveness，容易把下游故障放大为上游重启风暴。",[10,78,79],{},"Readiness 是否检查数据库，则要看业务。若这个 API 的全部请求都必须依赖数据库，持续失去数据库访问能力时退出流量可能合理；若首页缓存、静态信息或降级接口仍可用，把所有实例一起摘掉可能损失更多可用性。健康信号必须对应实际服务能力。",[10,81,82],{},"还要避免一种反馈回路：实例过载→健康检查超时→实例被摘掉→剩余实例更过载。解决办法可能包括轻量独立的健康接口、请求队列上限、合理探针阈值与容量冗余，而不是一味缩短检测间隔。",[10,84,85],{},"探针自身也消耗资源。昂贵的数据库查询、完整依赖扫描或高频磁盘操作，不适合每秒在所有实例上执行。更稳妥的做法是让检测具有明确成本，把深入诊断留给另外的监控与排障流程。",[23,87,89],{"id":88},"requests-描述调度需求limits-限制运行消耗","requests 描述调度需求，limits 限制运行消耗",[10,91,92],{},"再看一个片段：",[94,95,100],"pre",{"className":96,"code":97,"language":98,"meta":99,"style":99},"language-yaml shiki shiki-themes github-dark","resources:\n  requests:\n    cpu: 250m\n    memory: 256Mi\n  limits:\n    cpu: \"1\"\n    memory: 512Mi\n","yaml","",[49,101,102,114,122,135,146,154,164],{"__ignoreMap":99},[30,103,106,110],{"class":104,"line":105},"line",1,[30,107,109],{"class":108},"s4JwU","resources",[30,111,113],{"class":112},"s95oV",":\n",[30,115,117,120],{"class":104,"line":116},2,[30,118,119],{"class":108},"  requests",[30,121,113],{"class":112},[30,123,125,128,131],{"class":104,"line":124},3,[30,126,127],{"class":108},"    cpu",[30,129,130],{"class":112},": ",[30,132,134],{"class":133},"sU2Wk","250m\n",[30,136,138,141,143],{"class":104,"line":137},4,[30,139,140],{"class":108},"    memory",[30,142,130],{"class":112},[30,144,145],{"class":133},"256Mi\n",[30,147,149,152],{"class":104,"line":148},5,[30,150,151],{"class":108},"  limits",[30,153,113],{"class":112},[30,155,157,159,161],{"class":104,"line":156},6,[30,158,127],{"class":108},[30,160,130],{"class":112},[30,162,163],{"class":133},"\"1\"\n",[30,165,167,169,171],{"class":104,"line":166},7,[30,168,140],{"class":108},[30,170,130],{"class":112},[30,172,173],{"class":133},"512Mi\n",[10,175,176,179,180,183,184],{},[49,177,178],{},"250m"," 是四分之一个 CPU 单位；",[49,181,182],{},"256Mi"," 是二百五十六 mebibytes。调度器利用 requests 判断节点是否有足够可分配容量。容器在有资源可用时，可以使用超过 request 的资源；request 不等于消耗上限。",[30,185,186],{},"2",[10,188,189],{},"在常见 Linux 节点上，CPU limit 主要通过节流限制使用量，内存 limit 则与内核的内存限制及 OOM 处理有关。因此，CPU 超限更可能表现为延迟增长，内存问题则可能表现为进程被杀死和重启。不能用“资源不够”四个字代替具体机制。",[10,191,192],{},"如果把 API 的 CPU limit 设得很低，流量上涨后它也许仍显示 Running，但请求变慢。增加线程数不一定有帮助：所有线程仍在争抢同一份 CPU 时间预算。",[10,194,195],{},"如果内存逐渐上涨并最终 OOM，调高 limit 可以延后事故，却未必解决泄漏或无界缓存。先看内存变化形态、并发量与对象保留原因，再决定修程序、限制队列还是调整预算。",[10,197,198],{},"资源配置也有成本。request 过大，节点可能实际很空却无法再调度新 Pod；request 过小，多工作负载高峰重叠时可能争抢资源。初始值可以来自测量与保守估计，之后持续校准。",[23,200,202],{"id":201},"hpa-调整的是副本数","HPA 调整的是副本数",[10,204,205,206],{},"HorizontalPodAutoscaler 根据观测指标调整目标工作负载的副本数。使用 CPU 利用率百分比时，request 会参与分母：消耗 250m、request 500m，对应约50%的利用率；消耗不变，把 request 改成250m，比例就变成约100%。",[30,207,208],{},"3",[10,210,211],{},"这些判断还需要指标能被取得。CPU、内存资源指标通常由 Metrics Server 等实现通过资源指标 API 提供，并不是创建 HPA 对象就自动有了数据。使用自定义或外部指标时，还需要相应指标 API 的适配实现，并确认采集与授权链路正常。",[10,213,214],{},"这说明 request 的修改不仅影响调度，还可能改变扩缩容行为。若相关容器缺少必要的 CPU request，基于该资源利用率的计算也可能无法正常进行，不能简单把缺失理解为零。",[10,216,217,218,221],{},"以一个简化计算为例：当前两副本平均CPU利用率为90%，目标60%，按比例得到 ",[49,219,220],{},"2 × 90 / 60 = 3","，于是目标倾向于三个副本。实际控制器还要处理容忍区间、缺失指标、初始化中的 Pod、稳定窗口以及扩缩速率，不会把每次瞬时读数直接变成一次扩缩容。",[10,223,224],{},"HPA 也不负责让底层机器凭空出现。如果它把目标改成六个，节点只能容纳四个，另外两个仍可能 Pending。节点扩容是另一个控制过程，还涉及配额、启动时间与成本。",[10,226,227],{},[60,228,230],{"href":229},"/collections-assets/cloudnative/cn08-hpa-capacity.png",[64,231],{"alt":232,"src":229},"HPA 将副本目标提高到六个，调度器只能安排四个，另外两个 Pending；即使六个都运行，每实例二十条连接也会超过数据库八十条的预算。",[10,234,235],{},"扩容意图、节点容量和下游承载是三件事，副本增加不保证瓶颈消失。",[23,237,238],{"id":238},"选择能反映瓶颈的指标",[10,240,241],{},"异步摘要任务的CPU可能不高，因为大部分时间都在等待外部服务。队列却一直增长。如果只根据CPU扩容，控制器可能认为系统很闲。",[10,243,244],{},"这类场景可以考虑队列长度、最老任务等待时间或按副本分摊的待处理量。但使用这些指标之前，要弄清下游是否允许增加并发。若外部服务只允许每秒十次调用，扩容一百个worker不会让额度增加，反而可能带来更多限流和重试。",[10,246,247],{},"同样，API 延迟增长可能来自数据库锁、连接池或CPU。只有当新增副本能增加真正的处理能力时，扩容才对症。弹性设计需要把瓶颈、指标与执行动作连起来。",[10,249,250],{},"还要计算反应时间：发现流量变化、采集指标、控制器决策、调度、拉取镜像和应用预热，都需要时间。突然出现的尖峰可能在新副本就绪前已经到达，因此最小副本数、缓存和限流仍然重要。",[23,252,253],{"id":253},"两个副本不一定有两个故障域",[10,255,256],{},"两个 Pod 如果在同一台节点上，节点故障会同时影响两者。跨节点放置改善节点故障下的韧性；跨可用区还涉及区域性故障，以及网络与存储的额外约束和成本。",[10,258,259],{},"这就是为什么“副本数”不能单独代表可靠性。还要看副本分布、剩余容量、关键依赖是否共享同一个故障点，以及失败后多久恢复。",[10,261,262,263,266],{},"PodDisruptionBudget 可以约束经由驱逐 API 的自愿中断，例如节点维护时允许同时失去多少实例。但它挡不住物理节点突然故障，也不覆盖所有删除路径；Deployment 的滚动更新要由发布策略控制。",[30,264,265],{},"4"," 将PDB设得过于严格，还可能让节点维护一直无法完成。",[10,268,269],{},"可靠性配置是一组相互制约的决策。为了用户能持续阅读笔记，需要同时保证实例健康、容量有余量、依赖不过载、变更速度受控。单独打开某个“高可用”开关，无法取代这套推理。",[23,271,272],{"id":272},"动手想一想",[274,275,277],"h3",{"id":276},"题目-1","题目 1",[10,279,280],{},"应用正常初始化需要四十秒，liveness 每五秒检查一次、连续三次失败就重启。为什么一直起不来，怎样改比较合理？",[274,282,283],{"id":283},"提示",[10,285,286],{},"健康检查正在把正常的初始化过程解释成运行期故障。",[274,288,289],{"id":289},"答案",[10,291,292],{},"容器在初始化完成之前就被重启，下一次又从头开始。可设置合适的 startup probe，让它覆盖合理的启动窗口，再让 liveness 负责运行期无法自恢复的故障。窗口应结合最慢正常启动时间和余量，而非无限放大。",[274,294,296],{"id":295},"题目-2","题目 2",[10,298,299],{},"容器消耗250m CPU，request为500m，HPA目标利用率60%。只把request改为250m，会发生什么变化？",[274,301,283],{"id":302},"提示-1",[10,304,305],{},"CPU利用率的分母是什么？",[274,307,289],{"id":308},"答案-1",[10,310,311],{},"同样消耗对应的利用率从约50%变为约100%，可能触发更多副本。实际扩缩结果还受指标采样、其他容器、稳定窗口和控制器行为影响。修改request需要同时评估调度与自动扩缩容。",[274,313,315],{"id":314},"题目-3","题目 3",[10,317,318],{},"已经设置两个副本和PDB，为什么一次节点故障仍然让服务短暂中断？",[274,320,283],{"id":321},"提示-2",[10,323,324],{},"查看副本是否集中在同一节点，以及PDB约束哪类中断。",[274,326,289],{"id":327},"答案-2",[10,329,330],{},"两个副本可能共用一个故障域；PDB主要约束遵守驱逐流程的自愿中断，不阻止节点突然故障。还应检查副本分布、替代实例的调度容量、启动时间及下游依赖。",[23,332,333],{"id":333},"参考资料",[335,336,337,346,353,360],"ol",{},[338,339,340],"li",{},[60,341,345],{"href":342,"rel":343},"https://kubernetes.io/docs/concepts/workloads/pods/probes/",[344],"nofollow","Liveness, Readiness, and Startup Probes",[338,347,348],{},[60,349,352],{"href":350,"rel":351},"https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/",[344],"Resource Management for Pods and Containers",[338,354,355],{},[60,356,359],{"href":357,"rel":358},"https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/",[344],"Horizontal Pod Autoscaling",[338,361,362],{},[60,363,366],{"href":364,"rel":365},"https://kubernetes.io/docs/concepts/workloads/pods/disruptions/",[344],"Disruptions",[368,369,370],"style",{},"html pre.shiki code .s4JwU, html code.shiki .s4JwU{--shiki-default:#85E89D}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":99,"searchDepth":124,"depth":124,"links":372},[373,374,375,376,377,378,379,390],{"id":25,"depth":116,"text":25},{"id":72,"depth":116,"text":73},{"id":88,"depth":116,"text":89},{"id":201,"depth":116,"text":202},{"id":238,"depth":116,"text":238},{"id":253,"depth":116,"text":253},{"id":272,"depth":116,"text":272,"children":380},[381,382,383,384,385,386,387,388,389],{"id":276,"depth":124,"text":277},{"id":283,"depth":124,"text":283},{"id":289,"depth":124,"text":289},{"id":295,"depth":124,"text":296},{"id":302,"depth":124,"text":283},{"id":308,"depth":124,"text":289},{"id":314,"depth":124,"text":315},{"id":321,"depth":124,"text":283},{"id":327,"depth":124,"text":289},{"id":333,"depth":116,"text":333},"用慢启动、连接池耗尽和突发流量三个场景，理解探针、资源声明、HPA 与故障域。","md",{"collection":394,"order":395,"slug":396},"云原生",8,"health-and-capacity",true,"/column/cloudnative/health-and-capacity",{"title":5,"description":391},"column/cloudnative/health-and-capacity","71DjxdP3EamMOnh97CrxDMMT1LTFnC3kZr_GBeDfQZE",1791935829015]