Understanding the Limitations of Kubernetes Readiness Probes During Rolling Updates

Key Insights on Readiness Probes

Deploying rolling updates in Kubernetes holds great promise for zero-downtime application releases. Properly executed, traffic gradually transitions from old pods to new ones, and health indicators remain positive throughout the process. However, this ideal scenario can quickly unravel if the health checks aren't finely tuned to match the needs of the running application.

Real-World Implications

Take for example a recent rolling update for a Kubernetes application handling 5G signaling. Despite all readiness probes indicating green lights, an alarming 18% surge in session establishment failures occurred during the update. The underlying issue? The newly launched pods were technically running, yet they weren't fully prepared to manage operational traffic.

The Mechanism of Readiness Probes

At their core, Kubernetes readiness probes are designed to address a critical scenario: newly launched pods may come up successfully but aren’t instantaneously able to handle incoming requests. These probes are predominantly HTTP-based checks that verify whether the application responds correctly when prompted. A typical readiness probe sends a GET request; a 200 status implies the pod is good to go and ready to accept traffic.

This method works exceptionally well for lightweight, stateless applications. However, the narrative shifts considerably when it’s a matter of stateful applications or those that utilize protocols beyond standard HTTP.

Identifying the Gaps

In telecommunications, for instance, components that communicate via protocols like SIP or GTP face a unique challenge. These systems do not merely respond to HTTP requests; they need to establish connections, register with other network components, and finalize various handshakes before they can effectively serve traffic. Naturally, these processes can take considerable time, sometimes extending beyond a minute.

During this critical window, which can occur despite an immediate successful HTTP probe response, Kubernetes inaccurately marks the pod as ready. As a result, despite appearing healthy on dashboards, the infrastructure experiences degradation in actual usage—errors rise quietly as requests are routed to pods that aren't appropriately primed.

Case Study Summary

In the earlier referenced 5G deployment, the readiness probe configuration was simplified to a basic HTTP check against a lightweight endpoint. While this returned a success status within seconds, it could not confirm whether the pod finished SIP registration. This mismatch resulted in traffic being directed to active but unqualified pods, culminating in a notable spike in session establishment failures during peak usage.

Implementing Protocol-Aware Readiness Probes

To mitigate these issues, a better approach involves the use of protocol-aware readiness probes that go beyond simple HTTP responses. Such a probe should inspect whether the application has successfully completed necessary integration steps. For the 5G service example, this meant replacing the HTTP probe with an ‘exec’ probe, incorporating a shell command that actively queries registration states before signaling readiness:

readinessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - "check-registration-status.sh && exit 0 || exit 1"
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 12

This approach ensured that the readiness state was contingent upon actual operational readiness rather than on quick HTTP responses.

Lessons for Various Workloads

While the telecom scenario provides valuable lessons, the implications extend across various types of applications. Any system that depends on established connections, requires upstream registration, or involves warm-up periods should adjust its readiness probes accordingly. This applies particularly to message-oriented middleware, database-backed applications, and gRPC services, where interaction via other protocols is an essential part of the lifecycle.

The Bottom Line on Deployment Readiness

Utilizing Kubernetes means having the ability to define what ‘ready’ is for each specific application requirement. The ready state reflected by the default HTTP probe is merely an entry point for discussion. In situations where the gap between ‘running’ and ‘ready’ widens—such as in stateful or protocol-dependent workloads—the potential for unnoticed failures during rolling updates increases dramatically.

It’s crucial to align the readiness probes with what the application genuinely needs to operate effectively in a production environment. If your traffic routing is based solely on an HTTP 200 response that lacks insight into the actual readiness state, you're setting the stage for deployment outcomes that might look perfect on the surface but actually hide significant operational issues.