$ cat kubernetes-probes-production.md

Kubernetes Probes in Production: Configuration, Failure Modes, and Common Mistakes

#secops #technology #linux

In the previous article we covered what the three probes are for: liveness, readiness, and startup. But knowing which probe to use solve only part of the problem. What’s left is to understand all those parameters you encounter when you try to configure probes properly: which probe mechanism to use, timing parameters (and their dangerous defaults), plus the handful of configurations that often cause outages. Lets give a deeper look in this article.

Part 1: Probe Types

Kubernetes probes support four different protocols for monitoring pod health, and while httpGet is the most common, it isn’t always the ideal choice for every scenario. Let us review all available options to see how they work.

• httpGet

Kubernetes issues an HTTP GET against the container. Any 2xx or 3xx status is a success; everything else fails. For an HTTP application this is almost always the correct choice, because if the HTTP server has problems, the probe notices right away.

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080

Note that the probe is sent by the kubelet from the node, not from inside the container, so it bypasses anything you configured for in-cluster traffic. Avoid the tempting anti-pattern of wrapping curl in an exec probe:

# don't do this
readinessProbe:
  exec:
    command: ["curl", "-f", "localhost:8080/health"]  

That forks a process and loads a binary on every check to do something the kubelet does natively, for free.

• tcpSocket

The kubelet just opens a TCP connection to the port. If the handshake completes, the probe passes.

livenessProbe:
  tcpSocket:
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

This is a weak signal, and deliberately so. A listening socket says nothing about whether the application behind it works: the accept queue can be full, the worker pool exhausted, the event loop deadlocked, every request failing and yet the handshake still succeeds. Use it for things that genuinely have no health endpoint, such as a database or a plain TCP service, and don’t mistake it for a health check.

• exec

The kubelet runs a command inside the container; exit code 0 passes, anything else fails.

readinessProbe:
  exec:
    command: ["/app/bin/check-ready"]

Useful when the application ships a purpose-built health command, or for things with no network surface at all (batch workers, sidecars). The production caveat is cost: this spawns a process inside the container on every single check. At periodSeconds: 5 across 100 Pods that is 72,000 process creations an hour, so the command must be genuinely small and not a shell script that shells out six more times.

• grpc

For gRPC services, Kubernetes can speak the standard gRPC health checking protocol directly:

livenessProbe:
  grpc:
    port: 8080
  periodSeconds: 10

This replaces the old pattern of shipping a grpc_health_probe binary in the image and calling it with an exec probe.

Mechanism	Use it for	Watch out for
httpGet	HTTP/REST services with health endpoints	Nothing; this is the default
grpc	gRPC services implementing the health protocol	Service must implement the standard
tcpSocket	Services with no health endpoint at all	An open port proves almost nothing
exec	Purpose-built health commands, non-network workloads	Process spawn cost on every check

Part 2: Timing Parameters

This is where most production problems actually come from, and the defaults deserve a look before anything else:

Field	Default	Notes
initialDelaySeconds	0	Probing starts immediately
periodSeconds	10	How often to check
timeoutSeconds	1	The one that bites people
failureThreshold	3	Consecutive failures before acting
successThreshold	1	Must be 1 for liveness and startup

That timeoutSeconds: 1 default is the single most common cause of mysterious restarts. If your health endpoint occasionally takes 1.2 seconds under GC pressure or a cold connection pool, the default silently fails the probe and if it’s a liveness probe, it restarts a container that was working fine.

initialDelaySeconds

How long the kubelet waits after container start before probing at all.

initialDelaySeconds: 30

Try to avoid it since it is a fixed, unconditional waiting time: set it to 30 and every Pod in every rollout, forever, waits 30 seconds even when it was ready in two. Worse, setting it large to accommodate a slow start (initialDelaySeconds: 300 is a real thing people write) tells Kubernetes to ignore a genuinely dead container for five minutes.

A startup probe does the same job adaptively, it keeps checking, so a fast start is detected immediately and a slow one still gets its full budget. If you have a startup probe, you almost never need initialDelaySeconds.

periodSeconds

How often the check runs. Readiness generally wants a shorter period than liveness, because reacting quickly to “stop sending me traffic” is cheap and reversible, while “restart this container” is not cheap nor reversible:

Readiness: every 5 seconds
Liveness:  every 10 seconds

timeoutSeconds

How long the check may take before it counts as a failure. Set it above the realistic p99 of the endpoint. Setting it very high has the opposite problem: timeoutSeconds: 60 on a liveness probe means a hung container can sit there for minutes before anyone notices.

If your endpoint takes an entire minute to respond to a health check, it requires an upstream fix; a proper probe should reply in a fraction of a second and if cannot complete in well under a second is simply trying to do too much.

failureThreshold

How many consecutive failures are required before Kubernetes acts.

Probe 1 -> FAIL
Probe 2 -> FAIL
Probe 3 -> FAIL
             |
             v
       Threshold reached -> act

The counter resets on any success, which is the whole point: it distinguishes a persistent fault from the transient packet loss, GC pause, or CPU contention that every real cluster produces continuously.

## Part 3: Readiness During Rollouts and Termination

Readiness is not only a failure-handling mechanism it runs constantly, and it governs every rolling update.

A Pod failing readiness is simply removed from the Service’s endpoints while continuing to run:

Service
   |
   +---- Pod 1 [Ready]    <- receives traffic
   |
   +---- Pod 2 [Not Ready] <- running, no traffic
   |
   +---- Pod 3 [Ready]    <- receives traffic

Not Ready is not Not Alive, and during a deploy that distinction is doing real work: the Deployment waits for each new Pod to pass readiness before it proceeds to the next, so an honest readiness probe is what prevents a rollout from shifting traffic onto instances that aren’t warm yet. A readiness probe that returns 200 the instant the process starts makes a rolling update look successful while it drops requests.

Termination is where this gets complicated for critical systems where zero-downtime deployments matter, and it’s worth being precise because the common advice is wrong. When a Pod is deleted, two things happen in parallel:

1. Traffic routing stops: The cluster starts removing the pod’s IP address from all load balancers, ingress controllers, and proxies (EndpointSlices).
2. Shutdown begins: The pod receives a SIGTERM signal telling the application to shut down, and any preStop hook starts running.

Updating every router and proxy across the cluster takes a few seconds because it happens asynchronously. During that brief window, incoming traffic keeps arriving at the pod, even though the application has already been told to shut down. If the app stops immediately, those incoming requests fail with errors.

So there is a window where the application has already received SIGTERM but traffic is still arriving. Failing your readiness probe on shutdown does not close that window since deletion already triggered the service removal; the race is in the propagation. What actually closes it is a short preStop delay that keeps the Pod serving while the removal spreads:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]

terminationGracePeriodSeconds: 30

The grace period is the total budget for preStop plus draining; overrun it and the container is SIGKILLed mid-request. Probes are one part of lifecycle management, not all of it.

Part 4: Configurations That Cause Outages

Probes that are too aggressive

periodSeconds: 1
timeoutSeconds: 1
failureThreshold: 1

This says: one slow response, and act. Production clusters produce transient slowness constantly GC pauses, CPU contention from a noisy neighbour, a node under memory pressure, ordinary packet loss and none of it means the application is broken. Aggressive thresholds on a liveness probe convert that background noise into restarts, which is how a mild latency incident becomes a crash loop.

Timeouts tuned too low

Covered above, but it’s the most common single misconfiguration: leaving timeoutSeconds at its default of 1 while the health endpoint’s p99 is well above that. The probe fails under load, which is precisely when you can least afford to lose replicas.

A startup budget smaller than the real startup time

periodSeconds × failureThreshold for the startup probe is a hard deadline. If it’s shorter than a cold start with migrations and an empty cache, the Pod is killed, restarted, and killed again presenting as CrashLoopBackOff with no error in the application logs.

The ones from part one, still true here

Three mistakes carry over from part one and are worth naming because they’re just as common at scale: checking external dependencies in liveness (restarting a Pod doesn’t fix a database), pointing all three probes at a single /health endpoint so they can’t express different answers, and making readiness too strict, so a degraded-but-useful Pod gets pulled from the Service over a dependency it could tolerate.

Part 5: Quick Checklist

Mechanism httpGet for HTTP, grpc for gRPC, tcpSocket only when there’s nothing better, exec only when it’s genuinely cheap.

Timing

• Is timeoutSeconds above the endpoint’s p99? (The default of 1 is rarely right.)
• Does periodSeconds × failureThreshold give acceptable detection time?
• Is the threshold tolerant of a transient network issue or a brief packet loss?
• Is the startup budget larger than the slowest realistic cold start?

Semantics

• Does liveness check only things a restart could fix?
• Does readiness answer “can I still serve traffic even if i am degradated state”?
• Are the three probes on separate endpoints?

Lifecycle

• Does readiness stay honest during a rolling update?
• Is there a preStop delay to cover endpoint propagation?
• Is terminationGracePeriodSeconds longer than preStop plus drain time?

Global

• If every replica fails this check simultaneously, what happens?

Kubernetes already provides all the tools we need to keep our applications healthy through probes; our only job is to configure them correctly.

As always, I hope you found both of these articles helpful! :).