In this post we will review how to deploy a new application version on Kubernetes without interrupting requests. We will combine readiness probes, rolling updates, and graceful shutdown, then check what happens while the old and new versions run together.
Assume we have an HTTP API behind a Kubernetes Service and an ingress. A request normally takes less than a second, but some requests take 20 seconds. We want to replace the application image while clients continue using the API.
Setting the deployment strategy to RollingUpdate is a good start. However it does not tell Kubernetes when the application has finished initialization, or tell the application how to finish an accepted request before exiting.
These decisions need to agree.
The Deployment Problem
An application can fail during deployment in several ways.
The new pod starts listening before loading its configuration. Kubernetes sends it a request, and the request fails because initialization is incomplete.
The old pod receives a termination signal and closes its connections immediately. A request that was almost complete now fails at the client.
The application waits for requests, but closes its database pool first. The connection remains open, but the handler can no longer finish its work.
In all these cases the Deployment can eventually report success. That tells us the desired pods became available; it does not prove that clients experienced no errors along the way.
Define When the Application Is Ready
Use separate endpoints for separate questions:
/live: Is the process healthy enough to keep running?/ready: Can this instance accept application traffic now?
The readiness endpoint should return HTTP 503 during initialization, HTTP 200 when the instance can serve requests, and HTTP 503 again when draining begins.
Opening the HTTP port is not sufficient. Depending on the service, initialization can include loading configuration, preparing shared storage access, or establishing the subscriptions needed to process requests.
Keep the readiness check cheap. It can inspect initialization and draining state plus carefully chosen dependency checks. An expensive database query on every probe creates another source of load.
Liveness should normally remain independent of a temporary database or messaging outage. Restarting every application pod during a shared dependency failure can make recovery harder.
I introduced these probe concepts in an earlier NodeJS example: Create NodeJS Service with Kubernetes Liveness and Readiness Probes
Configure the Rolling Update
The following Deployment illustrates the relevant settings. Replace the image with your own application, implementing /live and /ready on port 8080. The container must include /bin/sleep for the preStop hook shown here.
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-api
spec:
replicas: 2
minReadySeconds: 10
progressDeadlineSeconds: 600
selector:
matchLabels:
app: example-api
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: example-api
spec:
terminationGracePeriodSeconds: 60
containers:
- name: api
image: registry.example.com/example-api:1.0.0
ports:
- name: http
containerPort: 8080
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
memory: "512Mi"
startupProbe:
httpGet:
path: /live
port: http
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 60
readinessProbe:
httpGet:
path: /ready
port: http
periodSeconds: 2
timeoutSeconds: 1
failureThreshold: 1
livenessProbe:
httpGet:
path: /live
port: http
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
lifecycle:
preStop:
exec:
command: ["/bin/sleep", "10"]
maxSurge: 1 permits one extra pod during the rollout. maxUnavailable: 0 prevents the Deployment controller from intentionally reducing available pods below the desired count during replacement. It does not prevent a pod from crashing or a node from failing.
minReadySeconds: 10 requires a new pod to remain ready for ten seconds before it counts as available. This helps avoid replacing an old pod immediately after a brief successful readiness check.
The startup probe gives the local health endpoint roughly five minutes to become healthy before repeated startup failures cause a restart. Readiness and liveness probes wait until startup succeeds. Here /live only establishes local health; /ready still gates application initialization afterward.
progressDeadlineSeconds makes a stalled rollout visible through the Deployment status. It does not automatically roll back a failed release.
The Service Must Select Ready Pods
The corresponding Service is simple:
apiVersion: v1
kind: Service
metadata:
name: example-api
spec:
selector:
app: example-api
ports:
- port: 80
targetPort: http
Keep the default behavior of publishing ready endpoints. Enabling publishNotReadyAddresses for this client-facing Service would undermine the readiness boundary.
Readiness controls normal Service routing, but routing changes take time to reach all components. Existing connections can also outlive the endpoint update. This is why shutdown needs more than returning HTTP 503 from /ready.
Understand the Termination Timeline
During normal pod deletion, Kubernetes begins updating endpoint information and the kubelet starts container termination. These activities are asynchronous; they are not one perfectly ordered event.
The preStop hook runs before the application receives SIGTERM. In this example it sleeps for ten seconds while the old application continues serving. This gives Service routing and ingress endpoint changes time to propagate before the listener starts shutting down.
The application receives SIGTERM after the hook completes. It should then mark itself draining, stop admitting new work, and wait for work it already accepted.
The shutdown sequence is:
- Pod deletion begins; routing components start removing the endpoint.
- The
preStopdelay runs while the application can still serve late traffic. SIGTERMstarts application draining and graceful HTTP shutdown.- Accepted work finishes while its dependencies remain available.
- The application closes dependencies and exits.
Ten seconds is an example, not a universal propagation guarantee. Measure the behavior of your actual ingress, load balancer, and connection pools, then configure the delay accordingly.
The container entrypoint must deliver SIGTERM to the application. If a shell launches the executable, use exec or another arrangement that correctly forwards signals.
Drain Requests Before Closing Dependencies
The HTTP server should stop accepting new connections and allow accepted requests to finish within a configured timeout.
For Go applications, http.Server.Shutdown provides this behavior for ordinary HTTP connections. Keep the process alive until Shutdown returns. ListenAndServe returning does not mean all handlers have completed.
Use a fresh shutdown context with its own timeout. Reusing a context that was canceled by SIGTERM would make the shutdown deadline expire immediately.
If the timeout expires, follow an explicit policy for remaining connections. Calling Shutdown alone does not forcibly close every active connection when its context expires.
Keep the database pool, messaging clients, and other handler dependencies open until the work using them has finished. Closing them first defeats graceful shutdown.
Detached goroutines and message consumers need their own admission and completion tracking. http.Server.Shutdown does not wait for every background operation in the process. WebSockets and other hijacked connections also need explicit shutdown handling.
For jobs that must survive process replacement, persist acceptance and provide recovery rather than relying on a long termination timeout. I covered one approach in: Build a PostgreSQL Job Queue with SKIP LOCKED
Budget the Whole Shutdown
terminationGracePeriodSeconds includes the preStop hook. The countdown does not start after the sleep finishes.
For this example, assume the shutdown phases run sequentially:
- Endpoint propagation delay: 10 seconds
- HTTP request drain timeout: 30 seconds
- Other cleanup and exit margin: 20 seconds
- Total termination grace: 60 seconds
If background work has another independent 30-second drain phase afterward, this budget must grow or the phases must be coordinated differently. Include every shutdown phase that can delay exit.
After the grace period expires, Kubernetes can forcibly terminate the remaining process. Cleanup code cannot finish after SIGKILL.
Long streams need an explicit contract too. A finite shutdown budget cannot preserve a connection that is expected to remain open forever. Define reconnection and recovery behavior for those clients.
Leave Capacity for the Replacement
A surge pod needs somewhere to run. If the cluster cannot schedule it, maxUnavailable: 0 can leave the rollout waiting while the old pods continue serving.
Reserve enough CPU, memory, pod slots, and network capacity for the temporary overlap. Remember that terminating pods can still consume resources while draining; actual resource usage can exceed the simple desired-replicas-plus-surge calculation.
The remaining serving capacity must also handle real traffic. A pod being Ready is not proof that it can absorb twice its normal load. Bound concurrent work and observe latency under the expected rollout conditions: Go Concurrency Limits: Prevent Goroutine Explosions with Backpressure
One steady replica can use maxSurge: 1 and maxUnavailable: 0 to get temporary overlap during a planned release. That still provides no redundancy if the only serving pod fails before its replacement is ready.
Where a PodDisruptionBudget Helps
For two application replicas, we might also configure:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: example-api
spec:
minAvailable: 1
selector:
matchLabels:
app: example-api
This can restrict voluntary evictions that use the eviction API, such as a normal node drain.
A PodDisruptionBudget does not control the Deployment's rolling update. maxUnavailable and maxSurge control that replacement behavior. It also cannot prevent involuntary node failures or direct pod deletion.
With one replica, minAvailable: 1 can block a voluntary eviction until another suitable pod is available. It does not create the replacement for you.
Make Old and New Versions Compatible
Rolling updates deliberately run different versions at the same time.
Both versions must understand the shared data and messages used during that overlap. A renamed database column can break old pods even when every readiness probe passes.
Use an expand-and-contract migration: add the new schema in a compatible form, deploy code that can work through the transition, migrate data if needed, and remove the old representation only after older code no longer needs it. Include the rollback window in this decision.
Sessions, download results, and accepted jobs must also remain accessible after a request moves to another pod. Pod-local memory or files cannot provide that by themselves.
An application-only release should avoid unnecessary database or broker restarts. Those components have their own availability and upgrade requirements. For a related database deployment example: Deploy a PostgreSQL Cluster on Kubernetes
Verify Through the Public Endpoint
Run representative traffic through the same ingress or load balancer clients use, then update the Deployment image. A direct request to a pod skips part of the behavior we need to verify.
Include short requests, requests that remain active across termination, and persistent connections. Use enough concurrent traffic to reveal routing gaps and capacity problems. Record errors, timeouts, and latency rather than checking only whether the rollout command succeeds.
Watch the rollout with:
kubectl rollout status deployment/example-api --timeout=10m
kubectl get pods -l app=example-api -w
Check that uninitialized pods receive no application traffic, new pods become available before old ones are removed, and accepted requests complete before dependencies close.
Repeat with a replacement that never becomes ready. The rollout should stall visibly while the existing available pods remain. Also test a request that exceeds the drain timeout, so the forced termination policy is understood before it happens during a release.
If the test client retries automatically, record its first-attempt failures as well. A successful retry can hide an interrupted request, and repeating a write may duplicate its effect without idempotency.
Kubernetes logs can help connect these client results to pod initialization and shutdown events: Collecting Kubernetes Logs using LOKI
Final Note
A reliable rolling deployment needs agreement between the application and Kubernetes.
Readiness keeps new instances out of traffic until they can serve. Rolling update settings preserve available capacity during replacement. Graceful shutdown lets old instances complete accepted work before their dependencies disappear.
Combine these with compatible application versions, enough temporary capacity, and a rollout test through the real client path. That is how we establish whether a deployment actually avoids interrupted requests.





