Full Blog TOC

Full Blog Table Of Content with Keywords Available HERE

Wednesday, November 18, 2026

Zero-Downtime Kubernetes Deployments: Readiness, Rolling Updates, and Graceful Shutdown



In this post we will review how to deploy a new application version on Kubernetes without interrupting requests. We will combine readiness probes, rolling updates, and graceful shutdown, then check what happens while the old and new versions run together.

Assume we have an HTTP API behind a Kubernetes Service and an ingress. A request normally takes less than a second, but some requests take 20 seconds. We want to replace the application image while clients continue using the API.

Setting the deployment strategy to RollingUpdate is a good start. However it does not tell Kubernetes when the application has finished initialization, or tell the application how to finish an accepted request before exiting.

These decisions need to agree.

The Deployment Problem

An application can fail during deployment in several ways.

The new pod starts listening before loading its configuration. Kubernetes sends it a request, and the request fails because initialization is incomplete.

The old pod receives a termination signal and closes its connections immediately. A request that was almost complete now fails at the client.

The application waits for requests, but closes its database pool first. The connection remains open, but the handler can no longer finish its work.

In all these cases the Deployment can eventually report success. That tells us the desired pods became available; it does not prove that clients experienced no errors along the way.

Define When the Application Is Ready

Use separate endpoints for separate questions:

  • /live: Is the process healthy enough to keep running?
  • /ready: Can this instance accept application traffic now?

The readiness endpoint should return HTTP 503 during initialization, HTTP 200 when the instance can serve requests, and HTTP 503 again when draining begins.

Opening the HTTP port is not sufficient. Depending on the service, initialization can include loading configuration, preparing shared storage access, or establishing the subscriptions needed to process requests.

Keep the readiness check cheap. It can inspect initialization and draining state plus carefully chosen dependency checks. An expensive database query on every probe creates another source of load.

Liveness should normally remain independent of a temporary database or messaging outage. Restarting every application pod during a shared dependency failure can make recovery harder.

I introduced these probe concepts in an earlier NodeJS example: Create NodeJS Service with Kubernetes Liveness and Readiness Probes

Configure the Rolling Update

The following Deployment illustrates the relevant settings. Replace the image with your own application, implementing /live and /ready on port 8080. The container must include /bin/sleep for the preStop hook shown here.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-api
spec:
  replicas: 2
  minReadySeconds: 10
  progressDeadlineSeconds: 600
  selector:
    matchLabels:
      app: example-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: example-api
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: api
          image: registry.example.com/example-api:1.0.0
          ports:
            - name: http
              containerPort: 8080
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              memory: "512Mi"
          startupProbe:
            httpGet:
              path: /live
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 60
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            periodSeconds: 2
            timeoutSeconds: 1
            failureThreshold: 1
          livenessProbe:
            httpGet:
              path: /live
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sleep", "10"]

maxSurge: 1 permits one extra pod during the rollout. maxUnavailable: 0 prevents the Deployment controller from intentionally reducing available pods below the desired count during replacement. It does not prevent a pod from crashing or a node from failing.

minReadySeconds: 10 requires a new pod to remain ready for ten seconds before it counts as available. This helps avoid replacing an old pod immediately after a brief successful readiness check.

The startup probe gives the local health endpoint roughly five minutes to become healthy before repeated startup failures cause a restart. Readiness and liveness probes wait until startup succeeds. Here /live only establishes local health; /ready still gates application initialization afterward.

progressDeadlineSeconds makes a stalled rollout visible through the Deployment status. It does not automatically roll back a failed release.

The Service Must Select Ready Pods

The corresponding Service is simple:

apiVersion: v1
kind: Service
metadata:
  name: example-api
spec:
  selector:
    app: example-api
  ports:
    - port: 80
      targetPort: http

Keep the default behavior of publishing ready endpoints. Enabling publishNotReadyAddresses for this client-facing Service would undermine the readiness boundary.

Readiness controls normal Service routing, but routing changes take time to reach all components. Existing connections can also outlive the endpoint update. This is why shutdown needs more than returning HTTP 503 from /ready.

Understand the Termination Timeline

During normal pod deletion, Kubernetes begins updating endpoint information and the kubelet starts container termination. These activities are asynchronous; they are not one perfectly ordered event.

The preStop hook runs before the application receives SIGTERM. In this example it sleeps for ten seconds while the old application continues serving. This gives Service routing and ingress endpoint changes time to propagate before the listener starts shutting down.

The application receives SIGTERM after the hook completes. It should then mark itself draining, stop admitting new work, and wait for work it already accepted.

The shutdown sequence is:

  1. Pod deletion begins; routing components start removing the endpoint.
  2. The preStop delay runs while the application can still serve late traffic.
  3. SIGTERM starts application draining and graceful HTTP shutdown.
  4. Accepted work finishes while its dependencies remain available.
  5. The application closes dependencies and exits.

Ten seconds is an example, not a universal propagation guarantee. Measure the behavior of your actual ingress, load balancer, and connection pools, then configure the delay accordingly.

The container entrypoint must deliver SIGTERM to the application. If a shell launches the executable, use exec or another arrangement that correctly forwards signals.

Drain Requests Before Closing Dependencies

The HTTP server should stop accepting new connections and allow accepted requests to finish within a configured timeout.

For Go applications, http.Server.Shutdown provides this behavior for ordinary HTTP connections. Keep the process alive until Shutdown returns. ListenAndServe returning does not mean all handlers have completed.

Use a fresh shutdown context with its own timeout. Reusing a context that was canceled by SIGTERM would make the shutdown deadline expire immediately.

If the timeout expires, follow an explicit policy for remaining connections. Calling Shutdown alone does not forcibly close every active connection when its context expires.

Keep the database pool, messaging clients, and other handler dependencies open until the work using them has finished. Closing them first defeats graceful shutdown.

Detached goroutines and message consumers need their own admission and completion tracking. http.Server.Shutdown does not wait for every background operation in the process. WebSockets and other hijacked connections also need explicit shutdown handling.

For jobs that must survive process replacement, persist acceptance and provide recovery rather than relying on a long termination timeout. I covered one approach in: Build a PostgreSQL Job Queue with SKIP LOCKED

Budget the Whole Shutdown

terminationGracePeriodSeconds includes the preStop hook. The countdown does not start after the sleep finishes.

For this example, assume the shutdown phases run sequentially:

  • Endpoint propagation delay: 10 seconds
  • HTTP request drain timeout: 30 seconds
  • Other cleanup and exit margin: 20 seconds
  • Total termination grace: 60 seconds

If background work has another independent 30-second drain phase afterward, this budget must grow or the phases must be coordinated differently. Include every shutdown phase that can delay exit.

After the grace period expires, Kubernetes can forcibly terminate the remaining process. Cleanup code cannot finish after SIGKILL.

Long streams need an explicit contract too. A finite shutdown budget cannot preserve a connection that is expected to remain open forever. Define reconnection and recovery behavior for those clients.

Leave Capacity for the Replacement

A surge pod needs somewhere to run. If the cluster cannot schedule it, maxUnavailable: 0 can leave the rollout waiting while the old pods continue serving.

Reserve enough CPU, memory, pod slots, and network capacity for the temporary overlap. Remember that terminating pods can still consume resources while draining; actual resource usage can exceed the simple desired-replicas-plus-surge calculation.

The remaining serving capacity must also handle real traffic. A pod being Ready is not proof that it can absorb twice its normal load. Bound concurrent work and observe latency under the expected rollout conditions: Go Concurrency Limits: Prevent Goroutine Explosions with Backpressure

One steady replica can use maxSurge: 1 and maxUnavailable: 0 to get temporary overlap during a planned release. That still provides no redundancy if the only serving pod fails before its replacement is ready.

Where a PodDisruptionBudget Helps

For two application replicas, we might also configure:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: example-api
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: example-api

This can restrict voluntary evictions that use the eviction API, such as a normal node drain.

A PodDisruptionBudget does not control the Deployment's rolling update. maxUnavailable and maxSurge control that replacement behavior. It also cannot prevent involuntary node failures or direct pod deletion.

With one replica, minAvailable: 1 can block a voluntary eviction until another suitable pod is available. It does not create the replacement for you.

Make Old and New Versions Compatible

Rolling updates deliberately run different versions at the same time.

Both versions must understand the shared data and messages used during that overlap. A renamed database column can break old pods even when every readiness probe passes.

Use an expand-and-contract migration: add the new schema in a compatible form, deploy code that can work through the transition, migrate data if needed, and remove the old representation only after older code no longer needs it. Include the rollback window in this decision.

Sessions, download results, and accepted jobs must also remain accessible after a request moves to another pod. Pod-local memory or files cannot provide that by themselves.

An application-only release should avoid unnecessary database or broker restarts. Those components have their own availability and upgrade requirements. For a related database deployment example: Deploy a PostgreSQL Cluster on Kubernetes

Verify Through the Public Endpoint

Run representative traffic through the same ingress or load balancer clients use, then update the Deployment image. A direct request to a pod skips part of the behavior we need to verify.

Include short requests, requests that remain active across termination, and persistent connections. Use enough concurrent traffic to reveal routing gaps and capacity problems. Record errors, timeouts, and latency rather than checking only whether the rollout command succeeds.

Watch the rollout with:

kubectl rollout status deployment/example-api --timeout=10m
kubectl get pods -l app=example-api -w

Check that uninitialized pods receive no application traffic, new pods become available before old ones are removed, and accepted requests complete before dependencies close.

Repeat with a replacement that never becomes ready. The rollout should stall visibly while the existing available pods remain. Also test a request that exceeds the drain timeout, so the forced termination policy is understood before it happens during a release.

If the test client retries automatically, record its first-attempt failures as well. A successful retry can hide an interrupted request, and repeating a write may duplicate its effect without idempotency.

Kubernetes logs can help connect these client results to pod initialization and shutdown events: Collecting Kubernetes Logs using LOKI

Final Note

A reliable rolling deployment needs agreement between the application and Kubernetes.

Readiness keeps new instances out of traffic until they can serve. Rolling update settings preserve available capacity during replacement. Graceful shutdown lets old instances complete accepted work before their dependencies disappear.

Combine these with compatible application versions, enough temporary capacity, and a rollout test through the real client path. That is how we establish whether a deployment actually avoids interrupted requests.

Sunday, November 8, 2026

Limit HTTP Request Body Size in Go: Handle Chunked and Gzip Requests Safely

A stone spillway controls a powerful mountain river before it reaches a peaceful forest valley.



In this post we will review how to limit HTTP request body size in Go using http.MaxBytesReader. We will handle requests with a known Content-Length, requests whose length is unknown, and gzip requests that become much larger after decompression.

A small API can still receive a very large request. If the application reads that request into memory without a limit, one client can consume resources intended for many users. With enough concurrent requests, the result can be an out-of-memory failure and a restarting pod.

The important question is where we enforce the limit. Checking a header is useful, but the application must also limit the bytes it actually reads.

The Content-Length Problem

Assume our API accepts JSON documents up to one MiB, or 1,048,576 bytes.

A tempting implementation is:

if request.ContentLength > maxBodyBytes {
    http.Error(writer, "request body too large", http.StatusRequestEntityTooLarge)
    return
}

This can reject a known oversized request before parsing it. However it is not sufficient by itself.

An HTTP/1.1 client can send a body using chunked transfer encoding, without a Content-Length header. In Go, an unknown request body length is represented by ContentLength equal to -1. HTTP/2 requests can also arrive without a known length, although HTTP/2 does not use HTTP/1.1 chunked transfer encoding.

A negative length passes the check above. If we then call io.ReadAll(request.Body), the application continues allocating memory while the client sends data.

The limit must apply to the reader, including when the length is unknown.

Use http.MaxBytesReader

Go provides a standard library helper for this purpose:

request.Body = http.MaxBytesReader(writer, request.Body, maxBodyBytes)

The replacement reader reports a *http.MaxBytesError when reading would exceed the configured limit. We can identify that error with errors.As and return HTTP 413 Content Too Large.

Wrapping the reader does not load the body or check it immediately. The limit is enforced as downstream code reads it.

This distinction matters. If a handler reads only the first few bytes, it has not established that the entire body fits. If an endpoint requires a complete document, read and validate the complete bounded document before performing its business operation.

Also notice that io.LimitReader is a different tool. It returns EOF at its limit, which can make a truncated input look like a complete input. A size check built with it normally needs to allow one extra byte and explicitly detect overflow. MaxBytesReader provides the HTTP-specific error directly.

Gzip Changes What We Need to Count

Assume a client compresses a large JSON document before sending it. The compressed body could be small enough to pass an ingress limit while the decompressed document is far above the API's allowed size.

For a gzip request, Content-Length describes the compressed representation. It does not tell us how many bytes the application will receive from the gzip reader.

Go's HTTP server does not automatically decompress a request just because it has Content-Encoding: gzip. Our code or framework middleware must do that explicitly.

For an API that accepts compressed input, use two limits:

incoming body
    -> compressed/input byte limit
    -> gzip decompression, when requested
    -> decompressed byte limit
    -> parsing
    -> business operation

The input limit bounds bytes read before decompression. The decompressed limit bounds bytes delivered to the application after expansion. They can have different values.

For an uncompressed request, the same sequence works without the decompression step.

A Complete Go Example

The following server accepts one JSON document at POST /document. It supports an absent Content-Encoding or a single gzip encoding, and rejects other encodings with HTTP 415.

The example uses only the standard library. Save the following as main.go.

package main

import (
    "compress/gzip"
    "encoding/json"
    "errors"
    "flag"
    "io"
    "log"
    "net/http"
    "strings"
    "time"
)

type document struct {
    MessageText string `json:"message"`
}

type bodyLimits struct {
    inputBytes   int64
    decodedBytes int64
}

func (b *bodyLimits) readBody(
    writer http.ResponseWriter,
    request *http.Request,
) ([]byte, error) {
    request.Body = http.MaxBytesReader(writer, request.Body, b.inputBytes)
    defer request.Body.Close()

    var reader io.Reader = request.Body
    if strings.EqualFold(strings.TrimSpace(request.Header.Get("Content-Encoding")), "gzip") {
        compressed, err := gzip.NewReader(request.Body)
        if err != nil {
            return nil, err
        }
        defer compressed.Close()
        reader = compressed
    }

    decoded := http.MaxBytesReader(writer, io.NopCloser(reader), b.decodedBytes)
    defer decoded.Close()
    return io.ReadAll(decoded)
}

func writeReadError(
    writer http.ResponseWriter,
    err error,
) {
    var limitError *http.MaxBytesError
    if errors.As(err, &limitError) {
        http.Error(writer, "request body too large", http.StatusRequestEntityTooLarge)
        return
    }
    http.Error(writer, "cannot read request body", http.StatusBadRequest)
}

func (b *bodyLimits) handleDocument(
    writer http.ResponseWriter,
    request *http.Request,
) {
    if request.Method != http.MethodPost {
        writer.Header().Set("Allow", http.MethodPost)
        http.Error(writer, "method not allowed", http.StatusMethodNotAllowed)
        return
    }
    encodings := request.Header.Values("Content-Encoding")
    encoding := strings.TrimSpace(request.Header.Get("Content-Encoding"))
    if len(encodings) > 1 || (encoding != "" && !strings.EqualFold(encoding, "gzip")) {
        http.Error(writer, "unsupported content encoding", http.StatusUnsupportedMediaType)
        return
    }
    if request.ContentLength > b.inputBytes {
        http.Error(writer, "request body too large", http.StatusRequestEntityTooLarge)
        return
    }

    data, err := b.readBody(writer, request)
    if err != nil {
        writeReadError(writer, err)
        return
    }
    var value document
    if err := json.Unmarshal(data, &value); err != nil {
        http.Error(writer, "invalid JSON document", http.StatusBadRequest)
        return
    }

    // Perform the business operation only after reading and validating the body.
    writer.WriteHeader(http.StatusNoContent)
}

func main() {
    inputBytes := flag.Int64("max-input-bytes", 1<<20, "maximum input body bytes")
    decodedBytes := flag.Int64("max-decoded-bytes", 1<<20, "maximum decompressed body bytes")
    flag.Parse()
    if *inputBytes <= 0 || *decodedBytes <= 0 {
        log.Fatal("body limits must be positive")
    }
    limits := &bodyLimits{inputBytes: *inputBytes, decodedBytes: *decodedBytes}
    mux := http.NewServeMux()
    mux.HandleFunc("/document", limits.handleDocument)
    server := &http.Server{
        Addr:              "127.0.0.1:8080",
        Handler:           mux,
        ReadHeaderTimeout: 5 * time.Second,
        ReadTimeout:       15 * time.Second,
        WriteTimeout:      15 * time.Second,
        IdleTimeout:       60 * time.Second,
    }
    log.Fatal(server.ListenAndServe())
}

Run the server:

go run main.go -max-input-bytes 1048576 -max-decoded-bytes 1048576

The Content-Length check is an early rejection for the input limit. The readers enforce the actual limits for known and unknown lengths.

Reading through the decompressed reader to EOF also checks that a gzip stream finishes correctly. A malformed or truncated stream returns HTTP 400. If either reader reports MaxBytesError, the response is HTTP 413.

json.Unmarshal checks the complete buffered document, so a valid JSON object followed by another object or unexpected trailing data is rejected. Ordinary trailing JSON whitespace is accepted, but still counts toward the byte limit.

Notice that the handler does not send success before reading the body. Once a response has started, changing its status may be too late. This is also discussed in HTTP Streaming Error Handling in GO: Why HTTP 200 Is Not Enough

Try Small and Oversized Requests

A small document should return HTTP 204:

curl -i -H 'Content-Type: application/json' \
    --data-binary '{"message":"hello"}' \
    http://127.0.0.1:8080/document

Create a JSON document with a two-MiB message:

python3 - <<'PY'
import json
with open("large.json", "w") as output:
    json.dump({"message": "x" * (2 * 1024 * 1024)}, output)
PY

Send it with its known length:

curl -i -H 'Content-Type: application/json' \
    --data-binary @large.json \
    http://127.0.0.1:8080/document

The server should return HTTP 413. To exercise HTTP/1.1 chunked input, send the same document without Content-Length:

curl --http1.1 -i \
    -H 'Content-Type: application/json' \
    -H 'Transfer-Encoding: chunked' -H 'Content-Length:' \
    --data-binary @large.json \
    http://127.0.0.1:8080/document

The reader still rejects it. Chunk boundaries do not reset the byte counter.

Now compress the document:

gzip -c large.json > large.json.gz
wc -c large.json large.json.gz

Send the compressed version:

curl -i -H 'Content-Type: application/json' \
    -H 'Content-Encoding: gzip' \
    --data-binary @large.json.gz \
    http://127.0.0.1:8080/document

Although this repetitive document compresses very well, the decompressed limit should still produce HTTP 413.

Middleware Order and Error Handling

In a framework such as Echo, place the input limit before request decompression and the decoded limit after it. Check the framework's actual execution order: middleware registration order alone can be misleading when wrappers are nested.

Both limits must run before body logging, parsing, or any other component that buffers the body. A limiter installed after io.ReadAll cannot undo the allocation.

Also make sure size errors survive the framework's error handling. A parser that converts every read error into a string can lose the MaxBytesError type. The application might then return HTTP 400 or 500 instead of 413.

Preserve wrapped errors where possible. If an existing abstraction cannot preserve them, the limiting reader can record that overflow occurred, and the handler boundary can inspect that state before writing its response. Do not identify overflow by matching an error message.

The sample keeps error handling directly beside the read, so it does not need that extra state.

A Body Limit Is One Part of the Memory Budget

The example buffers a bounded body for simplicity. This does not mean that one MiB of accepted JSON requires only one MiB of memory.

The buffer can have spare capacity. JSON decoding creates additional values, and request handling retains other state. Multiple simultaneous requests multiply those costs.

Keep the maximum request size configurable, choose limits for each endpoint's real needs, and also bound active requests and waiting work. I covered that part in Go Concurrency Limits: Prevent Goroutine Explosions with Backpressure

Streaming parsing can reduce buffering for large documents, but it still needs byte limits, validation, and a clear decision about when side effects are allowed. One successful json.Decoder.Decode call does not prove the body ended; a single-document API must also check what follows.

If you need to understand which JSON fields consume the space, see JSON Size Analysis

Size Limits and Deployment Limits

An ingress or reverse proxy body limit is useful because it can reject traffic before it reaches the service. Keep application limits as well: internal callers may reach the service directly, and a proxy might only see the compressed representation.

A byte limit does not impose a time limit. A client can send an allowed body very slowly. The example includes server timeouts; production values should reflect expected request sizes, upload speeds, and proxy behavior. Decompression also consumes CPU, so byte limits do not replace concurrency control or protection at the network edge.

For multipart uploads, apply an overall body limit before parsing, then add the required file and field limits. A multipart memory threshold alone is not a total upload-size limit; parsers can spill file data to disk.

Monitor 413 responses by endpoint and check that valid clients still fit. Avoid logging complete rejected bodies, which creates another resource problem and can expose sensitive data.

Verify the Boundaries

Before using the limit, check an empty body, valid small JSON, malformed JSON, and valid JSON followed by a second document. Check a body exactly at the limit and another one byte above it.

Repeat the size cases with known lengths, unknown lengths, and gzip input. Check truncated gzip data and unsupported Content-Encoding values. For oversized input, verify HTTP 413 and verify that the business operation never runs.

If a reader or middleware changes, repeat these checks through the actual handler chain. Testing a size helper in isolation will not catch a decompression layer or error wrapper installed in the wrong place.

Final Note

Use Content-Length for an early rejection, then enforce the real limit while reading. If requests can be compressed, bound both the incoming representation and the decompressed content.

Read and validate the required document before performing its operation, preserve size errors through middleware, and combine request limits with concurrency and time limits. This gives an API a predictable response to oversized input instead of an unpredictable memory failure.

Monday, November 2, 2026

Atomic File Writes in Go: Prevent Corrupted Files After a Crash

A complete replacement document is prepared beside protected storage, illustrating atomic file replacement in Go.



In this post we will review how to save a file in Go without leaving a partially written version when the process crashes. We will use a temporary file and an atomic rename, then explain the additional synchronization needed when the entire machine fails.

This is useful for JSON state files, configuration snapshots, and checkpoints. A file can be small and still be important: if the application cannot read it after restarting, a failed save can become a startup failure.

The examples target Linux and a local filesystem that supports atomic replacement and directory synchronization. The application owns the destination directory, and one writer controls the state being saved.

The Overwrite Problem

Assume our application stores its last processed record in state.json:

{"last_processed_id":100}

After more work, it needs to save 200. The obvious implementation is:

err := os.WriteFile("state.json", data, 0o600)

For an existing file, os.WriteFile truncates it before writing the replacement. If the process stops between these operations, the original content is already gone. If writing fails partway through, the file can contain incomplete JSON.

Checking the returned error is necessary, but it cannot restore the previous file. A sudden process termination also gives our code no opportunity to handle the error.

The Go documentation describes this partial-write behavior.

Reproduce an Interrupted Save

Create a disposable directory and run the examples there. First create a valid state file:

printf '{"last_processed_id":100}\n' > state.json

Save the following program as unsafe.go:

package main

import (
	"log"
	"os"
)

func main() {
	file, err := os.Create("state.json")
	if err != nil {
		log.Fatal(err)
	}
	if _, err := file.WriteString("{\"last_processed_id\":"); err != nil {
		log.Fatal(err)
	}
	os.Exit(1)
}

Run it:

go run unsafe.go
cat state.json

The program deliberately exits with an error after writing only the beginning of the document. The file now contains:

{"last_processed_id":

This is a controlled demonstration of an interrupted overwrite. It does not simulate a power failure, but it makes the application-level problem reproducible without waiting for a real crash.

Notice that defer would not fix this. Deferred functions do not run when the process is killed with SIGKILL or exits through os.Exit.

Prepare the New File Before Replacing the Old One

Instead of changing the current file, prepare its replacement separately:

  1. Serialize the complete state in memory.
  2. Create a uniquely named temporary file in the destination directory.
  3. Write the new content to that file.
  4. Synchronize and close it.
  5. Rename it over the destination.
  6. Synchronize the directory.

Until the rename, state.json remains untouched. On the Linux filesystem assumed here, a reader opening that path sees the old complete file or the new complete file. A reader that already has the old file open can continue reading it.

The temporary file must be on the same filesystem as the destination. Creating it in the destination directory is the simplest way to arrange this. Using the default temporary directory could put it on another filesystem, where the rename fails.

Do not delete the destination before renaming. That would introduce a period where the file is missing and discard the old copy before replacement succeeds.

The Linux rename documentation describes the replacement behavior.

Atomic Replacement and Durability

There are two separate requirements here.

Atomic replacement controls what readers observe while the system is running. They should never open a half-written replacement through the destination path.

Durability concerns what survives a machine crash or power loss. A successful write or close can leave data in operating-system caches. A rename alone does not establish that both the new content and its directory entry reached persistent storage.

The Linux sequence used here is:

write temporary file
sync temporary file
close temporary file
rename temporary file to destination
sync destination directory

The file sync requests persistence of its content and metadata. The directory sync requests persistence of the name change. Both operations can fail, and their errors must be handled.

This is the standard persistence sequence for this Linux use case, subject to the filesystem and storage stack honoring these operations. It does not protect against failing hardware or replace backups.

The Linux fsync documentation explains why the directory requires its own synchronization.

Go Implementation

The following program uses only the standard library and requires Go 1.20 or later because it uses errors.Join. Save it as main.go and run it separately from unsafe.go:

package main

import (
	"errors"
	"fmt"
	"log"
	"os"
	"path/filepath"
)

func writeFileAtomic(
	filePath string,
	data []byte,
) error {
	directory := filepath.Dir(filePath)
	temporary, err := os.CreateTemp(directory, ".state-*.tmp")
	if err != nil {
		return fmt.Errorf("create temporary file: %w", err)
	}
	defer temporary.Close()
	defer func() {
		err := os.Remove(temporary.Name())
		if err != nil && !errors.Is(err, os.ErrNotExist) {
			log.Printf("remove temporary file: %v", err)
		}
	}()

	if _, err := temporary.Write(data); err != nil {
		return fmt.Errorf("write temporary file: %w", err)
	}
	if err := temporary.Sync(); err != nil {
		return fmt.Errorf("sync temporary file: %w", err)
	}
	if err := temporary.Close(); err != nil {
		return fmt.Errorf("close temporary file: %w", err)
	}
	if err := os.Rename(temporary.Name(), filePath); err != nil {
		return fmt.Errorf("replace destination: %w", err)
	}
	if err := syncDirectory(directory); err != nil {
		return fmt.Errorf("destination replaced, directory sync failed: %w", err)
	}
	return nil
}

func syncDirectory(
	directory string,
) error {
	file, err := os.Open(directory)
	if err != nil {
		return err
	}
	syncError := file.Sync()
	closeError := file.Close()
	return errors.Join(syncError, closeError)
}

func main() {
	data := []byte("{\"last_processed_id\":200}\n")
	if err := writeFileAtomic("state.json", data); err != nil {
		log.Fatal(err)
	}
	fmt.Println("Saved state.json")
}

Run the safe version:

go run main.go
cat state.json

The destination now contains:

{"last_processed_id":200}

os.CreateTemp creates a unique file with restrictive permissions, 0o600 before applying the process umask. This example retains those permissions when the file becomes state.json.

The deferred close is a fallback for early failures. The explicit close before rename is checked. Deferred removal cleans up an unpublished temporary file during a normal return; after a successful rename, the temporary pathname no longer exists.

The helper expects the parent directory to exist and remain stable during the operation. Create the application's state directory during initialization. Persisting a newly created directory hierarchy requires additional synchronization of its parent directories.

What Happens at Each Failure Point?

If the temporary write, file sync, or close fails, the rename does not happen. An existing destination remains unchanged.

If the process stops before rename, an existing destination still contains the old version. A temporary file may remain because deferred cleanup did not run.

If rename succeeds, new opens of the destination see the replacement. If the process then stops before directory synchronization, its persistence across a later machine failure has not been established.

If directory synchronization fails, the function returns an error that explicitly says the destination was already replaced. We cannot treat every error as meaning that nothing changed. The new version is visible, but its durability was not confirmed.

On the first save there is no old version to preserve. A process interruption before rename leaves the destination absent, possibly alongside an unfinished temporary file.

These distinctions matter when deciding whether to retry or report success. A checkpoint should also be coordinated with the work it represents; safely writing the file does not make an external side effect and the checkpoint one transaction.

The same uncertainty appears when a connection breaks after a server has performed its work, as discussed in HTTP Streaming Error Handling in GO: Why HTTP 200 Is Not Enough.

Verify the Failure Paths

Reset state.json to 100. In a disposable copy of main.go, insert os.Exit(1) immediately before os.Rename and run it again.

state.json should still contain 100. The completed temporary file should remain beside it. Repeat with an exit after writing only part of the temporary content: the destination should still be unchanged.

Remove the injected exit and run the normal version. The destination should contain 200, and the operation should leave no temporary file behind.

Also check a failed rename and a failed directory sync. The first should preserve the existing destination. The second should report an error while leaving the new content visible.

These checks verify the process-interruption behavior. They do not prove recovery after a power loss; that requires testing the actual filesystem and storage environment with an appropriate crash-testing setup.

Permissions and Concurrent Writers

Replacing a file installs a new filesystem object. It does not automatically preserve the old file's permissions, ownership, ACLs, or extended attributes. If specific metadata is required, apply it to the temporary file before its sync and publication.

The example also assumes the destination is a regular state-file path, not a symbolic link whose target must be updated. Renaming onto a symbolic link replaces that link itself.

Unique temporary names prevent writers from sharing the same temporary file. They do not prevent lost updates.

Assume two workers read 100, independently calculate 101, and each publish a complete file. Both files are valid, but one update is lost. Within one process, serialize the entire read-modify-write operation. Across processes, use appropriate coordination or a transactional database.

This is related to the distinction between individual atomic operations and a complete synchronized operation discussed in Go Atomic Value is NOT sync.

Operational Limits

A replacement temporarily needs space for both versions. Near a disk limit, even a small state update can fail during writing or synchronization. Keep the old version and surface the failure instead of continuing as if the state was saved.

Abrupt exits can leave temporary files. If startup cleanup removes them, ensure no active writer can still own them; a broad filename match alone is not sufficient.

Go does not promise atomic os.Rename behavior on every operating system. Windows, network filesystems, container volume drivers, and object-storage mounts require checking their own replacement and persistence semantics. Do not infer those guarantees from a successful test on a local Linux directory.

For several files that must change together, atomic replacement of each file separately is insufficient. Use a transactional store or a publication scheme that switches one reference to a complete version.

When the state already belongs in PostgreSQL, a database transaction can also keep a business update and its queued work together, as shown in Build a PostgreSQL Job Queue with SKIP LOCKED.

Final Note

For a small state file, the useful pattern is straightforward: prepare the complete replacement, sync and close it, publish it with a same-filesystem rename, then sync the directory when durability is required.

This keeps an interrupted save from destroying the last complete version and makes the remaining boundaries explicit: persistence, concurrent updates, and recovery after an uncertain result.

Sunday, October 25, 2026

Build a PostgreSQL Job Queue with SKIP LOCKED

PostgreSQL background queue distributing separate jobs to three workers while skipping a locked job.


In this post we will review how to use PostgreSQL as a background job queue. We will use FOR UPDATE SKIP LOCKED to let multiple workers claim jobs without waiting for the same row, and add leases so unfinished work can be recovered after a worker crashes.

Assume our application needs to generate reports. The HTTP request should save the report request and return quickly. Background workers will generate the reports independently.

If we already use PostgreSQL, a table can be enough to start. However a queue needs more than a SELECT followed by a DELETE. We need to decide who owns each job, when it can be retried, and what happens if the worker disappears.

The Job Claiming Problem

Let's assume two workers execute the same query:

SELECT job_id, payload
FROM background_job
WHERE status = 'queued'
ORDER BY job_id
LIMIT 1;

Both can receive the same job. Updating its status afterward does not fix the race unless the claim itself is coordinated.

We can lock the selected row using FOR UPDATE. However without SKIP LOCKED, another worker can wait for the first row even when other jobs are available.

SKIP LOCKED tells PostgreSQL to skip rows it cannot immediately lock. This makes it useful for distributing queue work among concurrent consumers. It is not appropriate for a normal query that needs a complete, consistent view of the matching data.

Create the Queue Table

The following examples use PostgreSQL SQL. The $1 and $2 values are parameters supplied by the application through its database driver.

CREATE TABLE background_job (
    job_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    payload jsonb NOT NULL,
    status text NOT NULL DEFAULT 'queued'
        CHECK (status IN ('queued', 'running', 'done', 'failed')),
    available_at timestamptz NOT NULL DEFAULT now(),
    lease_until timestamptz,
    lease_version bigint NOT NULL DEFAULT 0,
    attempts integer NOT NULL DEFAULT 0,
    created_at timestamptz NOT NULL DEFAULT now(),
    finished_at timestamptz
);
CREATE INDEX background_job_queued_idx
ON background_job (available_at, job_id)
WHERE status = 'queued';
CREATE INDEX background_job_running_idx
ON background_job (lease_until, job_id)
WHERE status = 'running';

The payload contains the information required to perform the job. In a real application I prefer keeping it small, for example a report ID and its parameters, rather than a large document.

available_at enables delayed retries. lease_until defines how long a worker owns a running job. lease_version changes every time the job is claimed, so an old worker cannot complete a newer worker's claim.

Insert a Job

INSERT INTO background_job (payload)
VALUES ('{"report_id": "report-123"}'::jsonb)
RETURNING job_id;

If a business change and its background job must both be saved, insert them in the same database transaction. For example, create the report request and its queue row together.

This prevents a failure between saving the request and scheduling the work. The worker can see the job after the transaction commits.

Claim a Job Atomically

The worker executes the following statement:

WITH candidate AS (
    SELECT job_id
    FROM background_job
    WHERE
        (status = 'queued'
            AND available_at <= statement_timestamp())
        OR
        (status = 'running'
            AND lease_until <= statement_timestamp())
    ORDER BY available_at, job_id
    LIMIT 1
    FOR UPDATE SKIP LOCKED
)
UPDATE background_job AS job
SET
    status = 'running',
    lease_until = statement_timestamp() + interval '60 seconds',
    lease_version = job.lease_version + 1,
    attempts = job.attempts + 1
FROM candidate
WHERE job.job_id = candidate.job_id
RETURNING
    job.job_id,
    job.payload,
    job.lease_version,
    job.attempts;

The selected row is locked and updated within one statement. Other workers skip it while the claim transaction holds the lock.

Notice that the query also accepts running jobs whose leases expired. This is how another worker recovers work after a crash. Each new claim increments lease_version, including a recovered claim.

If no eligible job is available, the statement returns no rows. The worker waits before polling again. Add some jitter to the polling delay so workers do not all wake up together.

Run this statement in autocommit mode, or commit its short transaction immediately. The claim must be committed before the worker starts the report generation.

Do Not Keep the Transaction Open While Working

A tempting implementation is to select and lock a job, generate the report, then commit.

This holds a database connection and a row lock throughout the slow operation. Long transactions also interfere with database maintenance and make failure handling harder.

Instead, the flow is:

  1. Claim the job and commit.
  2. Perform the work outside the claim transaction.
  3. Mark the job as completed using its claim version.

After step 1 the row lock is gone. The running status and lease now represent ownership. SKIP LOCKED coordinates the short claim; it does not protect the entire execution.

Complete Only the Current Claim

After generating the report, the worker executes:

UPDATE background_job
SET
    status = 'done',
    lease_until = NULL,
    finished_at = statement_timestamp()
WHERE
    job_id = $1
    AND lease_version = $2
    AND status = 'running'
    AND lease_until > statement_timestamp()
RETURNING job_id;

$1 is the job ID, and $2 is the lease version returned when this worker claimed it.

The application must check whether a row was returned. No row means that this completion was not accepted. The worker might have exceeded its lease, or another worker might already own a newer claim.

The lease check deliberately prevents completion after expiration, even if another worker has not reclaimed the job yet. Treat this as lost ownership, rather than silently reporting success.

Crash Recovery Does Not Mean Exactly Once

Assume the worker sends an email, then crashes before marking the job as done.

The lease expires. Another worker claims the job and sends the email again.

The queue cannot determine whether the first email was sent. This design provides at-least-once processing attempts, so the job handler must tolerate retries.

Use a stable idempotency key derived from the job or business operation. For an external API, use its idempotency support when available. For database changes, enforce a unique business key and commit the effect together with the guarded completion update when both live in the same database.

Do not use lease_version as the idempotency key. It changes on every attempt, while the business operation must retain the same identity.

The version check protects the queue row from stale workers. It does not stop a stale worker from calling an external service. That requires cooperation from the destination, idempotency, or a stronger fencing mechanism supported by that destination.

Choose a Lease Duration

The example uses a 60-second lease to keep the SQL readable. Production lease durations and polling intervals should be configurable.

If a job can take longer than its lease, renew it periodically:

UPDATE background_job
SET lease_until = statement_timestamp() + interval '60 seconds'
WHERE
    job_id = $1
    AND lease_version = $2
    AND status = 'running'
    AND lease_until > statement_timestamp()
RETURNING job_id;

Again, no returned row means ownership was lost. Stop further work where possible and do not acknowledge the old claim. Cancellation cannot undo a request that already reached an external system.

Renew well before expiration and allow for scheduling delays and temporary database latency. A very short lease causes unnecessary duplicate attempts. A very long lease delays recovery after a crash.

Retries and Failed Jobs

For a retryable error, the current owner can release the job with a delay:

UPDATE background_job
SET
    status = 'queued',
    available_at = statement_timestamp() + interval '30 seconds',
    lease_until = NULL
WHERE
    job_id = $1
    AND lease_version = $2
    AND status = 'running'
    AND lease_until > statement_timestamp()
RETURNING job_id;

Use bounded exponential backoff with jitter in production, and distinguish transient failures from invalid jobs that will never succeed. Avoid recording secrets or entire sensitive payloads in error messages.

The example claim query intentionally has no retry limit. Before using it in production, choose a maximum number of attempts and move exhausted jobs to failed. That policy must cover expired leases as well as errors caught by a live worker; otherwise a repeatedly crashing job can be reclaimed forever.

For example, a cleanup process can mark eligible queued or expired running jobs as failed once attempts reaches the configured limit, while the claim query excludes those jobs. Never fail a job merely because its attempt count reached the limit while its current lease is still valid.

Keep failed jobs available for investigation and explicit replay. A replay must preserve the business operation's idempotency rules.

What About Processing Order?

ORDER BY determines the preference for claiming eligible jobs. With SKIP LOCKED, a locked earlier job can be skipped, and concurrent workers can finish in any order.

This is suitable for independent report requests. It is not enough when events from the same conversation or customer must be processed sequentially.

For per-conversation ordering, store a conversation key and a sequence value defined by the producer. Only the earliest unfinished job for a key should be eligible for claiming. A NOT EXISTS condition can exclude candidates that have an earlier unfinished job with the same key.

Keep the eligibility rule on lease recovery too. An expired first job must be retried before a later job becomes eligible.

Also decide what a permanently failed first job means: block the conversation for investigation, or explicitly allow later jobs to continue. This is a business decision, not something SKIP LOCKED resolves.

Notice that an identity column alone does not guarantee business event order or transaction commit order. If ordering matters, define it at ingestion rather than assuming the generated job IDs express it.

Bound the Workers and the Backlog

Each worker should claim only the work it can start. Do not claim thousands of jobs and leave them waiting in memory while their leases expire.

Use a fixed worker count and consider the total across all application replicas. Ten pods with twenty workers can execute two hundred jobs at the same time.

A database queue also needs a backlog policy. Persistent storage is finite. Bound payload sizes, monitor queued bytes and job count, and define admission behavior when the backlog becomes too large.

Keep transactions short and monitor slow claim queries. The OR condition for ready and expired jobs is convenient for this example, but its execution plan should be checked against realistic queue size and traffic. Larger queues may benefit from a separate bounded lease-recovery process and a simpler queued-job claim.

Completed rows must also be removed or archived in bounded batches. Frequent status updates and deletes create dead tuples, so monitor autovacuum, table growth, and index growth.

Measure and Verify

Useful measurements include queue depth, age of the oldest ready job, execution duration, retry count, expired leases, failed jobs, and database time spent claiming work.

The age of the oldest ready job is particularly useful. A queue can have a modest number of jobs while one important job waits for a long time.

Before relying on the queue, verify the failure paths:

  1. Two workers claiming concurrently receive different jobs.
  2. A locked candidate does not prevent another worker from claiming an eligible row.
  3. A worker crash leaves a job recoverable after lease expiration.
  4. A stale worker cannot complete, renew, or release a newer claim.
  5. Retrying after an external side effect does not repeat the business operation.
  6. An exhausted job eventually becomes failed, including after repeated crashes.
  7. If ordering is required, a later event cannot run while an earlier event remains unfinished.

When Is PostgreSQL Enough?

A PostgreSQL queue is useful when the application already depends on PostgreSQL, jobs fit its capacity, and creating a job in the same transaction as business data simplifies correctness.

It also means queue traffic competes with application queries for database resources. A dedicated broker can be a better choice for high message volume, fan-out, independent scaling, or messaging features that would otherwise require substantial custom code.

Final Note

FOR UPDATE SKIP LOCKED solves one specific problem: workers can claim different rows without waiting behind the same locked candidate.

A reliable background queue also needs short transactions, leases, ownership checks, idempotent handlers, bounded retries, and backlog monitoring. Once these rules are explicit, PostgreSQL can provide a practical starting point without adding another service to the deployment.


Thursday, October 22, 2026

Go Concurrency Limits: Prevent Goroutine Explosions with Backpressure

A bounded queue feeds four processing lanes through an admission gate, illustrating controlled concurrent work in Go.



In this post we will review how to limit concurrent work in GO, why a semaphore can still leave us with too many goroutines, and what should happen when the application cannot accept more work.

Goroutines make it easy to run operations concurrently. However starting a goroutine is not the same as having enough capacity to complete its work. Under sustained load, this difference can cause high memory usage, timeouts, and restarting pods.

Let's start with a simple example.

The Goroutine Explosion Problem

Assume our application receives 100 requests per second. Each request starts 10 independent checks, and each check takes two seconds.

At a steady rate, we can have approximately 2,000 checks in flight. If the dependency slows down and each check takes ten seconds, the number grows to approximately 10,000, assuming we keep accepting work at the same rate.

Each operation can retain more than its goroutine stack: request data, parsed objects, buffers, and references to other application state. The dependency might also be unable to handle so many concurrent calls.

This creates a feedback loop. A slower dependency leaves more operations in flight. The additional work increases pressure, which can make the dependency even slower.

A concurrency limit puts a boundary on this growth. However the location of the boundary matters.

The Semaphore Mistake

A buffered channel can act as a semaphore. Sending a value acquires a slot; receiving a value releases it.

A tempting implementation looks like this:

slots := make(chan struct{}, 10)

for _, item := range items {
    go func(item int) {
        slots <- struct{}{}
        defer func() { <-slots }()

        process(item)
    }(item)
}

This limits how many process calls can run concurrently. However it does not limit how many goroutines we create.

If items contains one million entries, the loop can still create one million goroutines. Most of them will wait for a slot while retaining their arguments and other reachable data.

We have limited the active work, but moved the backlog into waiting goroutines.

Acquire Before Starting the Goroutine

For a batch with one producer, the simple solution is to acquire the slot before starting the goroutine. The producer waits when all slots are occupied.

The following complete example uses only the standard library. Save it as main.go:

package main

import (
    "flag"
    "fmt"
    "sync"
    "time"
)

func main() {
    concurrency := flag.Int("concurrency", 10, "maximum concurrent tasks")
    count := flag.Int("count", 100, "number of tasks")
    flag.Parse()

    if *concurrency <= 0 || *count < 0 {
        fmt.Println("concurrency must be positive and count must not be negative")
        return
    }

    slots := make(chan struct{}, *concurrency)
    var workers sync.WaitGroup

    for item := 0; item < *count; item++ {
        slots <- struct{}{}
        workers.Add(1)

        go func(item int) {
            defer workers.Done()
            defer func() { <-slots }()

            process(item)
        }(item)
    }

    workers.Wait()
}

func process(item int) {
    time.Sleep(100 * time.Millisecond)
    fmt.Printf("processed item %d\n", item)
}

Run it using:

go run main.go -concurrency 10 -count 100

The sleep simulates a slow operation. In a real application, process could call an API or handle a database record.

The producer cannot keep creating task goroutines while all slots are occupied. At most ten process calls run concurrently with the default configuration, and the main goroutine waits before scheduling more work.

The deferred receive releases the slot when the worker returns. The WaitGroup makes the program wait for the scheduled workers before exiting. It does not limit concurrency by itself.

This example illustrates admission and cleanup. A production implementation also needs task error handling and cancellation. Completion order is not guaranteed.

What Backpressure Actually Means

Backpressure means that a downstream capacity limit slows or stops upstream production.

In the batch example, the producer waits for a slot. If it reads records incrementally from a database, it can stop reading ahead while the workers are busy.

However acquiring a slot before starting a goroutine does not automatically bound an entire HTTP service. The HTTP server already runs handlers concurrently. Thousands of handlers can wait for the same semaphore while retaining their request data.

For a service, we need an admission policy as well as an execution limit.

Bound Active Work and Waiting Work

There are two separate limits:

  • Active work: how many operations may execute concurrently.
  • Waiting work: how many accepted operations may wait to execute.

A worker pool with 20 workers and a queue of 100 jobs bounds those two parts of the system. However if producers block indefinitely while sending to the full queue, the backlog can still accumulate outside it.

When capacity is exhausted, choose an explicit behavior:

  • Wait for a bounded duration, provided the waiting callers are also bounded.
  • Reject the operation and let the caller decide whether to retry.
  • Save it to a durable queue for later processing.

For HTTP APIs, temporary service overload often maps to HTTP 503. A caller exceeding its rate allowance often maps to HTTP 429. The exact response should follow the API contract.

Do not silently drop work or report success for an operation that was never accepted.

Scope the Limit Correctly

A semaphore created separately for every request only limits each request. One hundred requests with ten slots each can still run 1,000 operations.

If the goal is protecting a shared dependency, the limiter should usually be shared across the code paths calling that dependency within the process.

Also notice that a process limit is not a cluster limit. Ten pods with a limit of 20 can issue approximately 200 concurrent calls. Scaling out can increase pressure on the dependency unless we account for the total.

Different resources may need different limits. Database writes and external model calls do not necessarily consume the same capacity.

Concurrency Limits Are Not Rate Limits

A concurrency limit controls how many operations are in flight. A rate limit controls how many operations may start during a period.

Twenty slots can permit thousands of fast operations per second. The same twenty slots might complete only a few operations per second when the dependency is slow.

If an external API allows 100 requests per second, a semaphore alone does not enforce that allowance. We may need both a rate limiter and a concurrency limiter.

Cancellation and Nested Work

A production task should not wait for a slot after its request has been canceled. Waiting for admission should observe the request context, and the operation itself should use cancellation and appropriate timeouts.

Canceling a context does not forcibly stop a goroutine. The code must observe it, and blocking dependencies must support cancellation or have their own timeouts.

Another subtle problem is nested parallelism. Suppose every parent task holds a slot, starts child tasks using the same limiter, and waits for them. If all slots belong to parents, the children cannot start and the parents cannot finish.

Avoid this circular wait. Depending on the design, run child work inline when another slot is unavailable, or schedule the work so a parent does not hold a scarce slot while waiting for children that need it.

Choosing the Limit

There is no concurrency value that works for every application.

Start with the resource we want to protect: CPU, memory, database connections, or the capacity of an external service. For CPU-heavy work, available CPU capacity matters. For operations that mostly wait on a network, a higher concurrency may help until another resource becomes the bottleneck.

A useful starting estimate is:

required concurrency ≈ target operations per second × average operation duration in seconds

For 100 operations per second taking 0.2 seconds each, the estimate is 20 operations in flight. This is a steady-state average, not a safe production setting by itself. Latency variation and bursts still matter.

Measure active tasks, queue depth, admission wait time, rejected tasks, task duration, and memory usage. Increasing a limit is useful only while throughput improves without unacceptable latency, errors, or resource consumption.

Verify the Overload Behavior

A useful check submits more work than the configured capacity and holds the active tasks behind a synchronization barrier. Count the active operations and verify that the maximum never exceeds the limit.

Then fill the waiting capacity and verify the defined behavior for the next operation. Check cancellation while waiting, slot release after a task error, and shutdown with work still outstanding.

Also test with a slow dependency. That is when an apparently harmless backlog often becomes visible.

Final Note

Goroutines are inexpensive, but the work they retain is not free.

Limit active execution, bound waiting work, and decide what happens when both capacities are full. With these boundaries, a traffic spike becomes controlled waiting or an explicit rejection instead of uncontrolled growth inside the application.

Wednesday, October 14, 2026

How I Get at Least 10x More Work Done with an AI Coding Agent



In this post we will review how I use an AI coding agent to get at least 10 times more development work done. The largest improvement comes from combining small tasks, reusable agent skills, and continuous review.

A feature rarely ends with writing a function. There are callers to update, configuration to connect, tests to adjust, deployment files to check, and sometimes a user interface to change. All these steps take time. An agent that can work inside the repository can handle much of this work while I concentrate on the decisions.

Looking back at my last two weeks of development and agent conversations, the interesting part is the variety: investigating failures, understanding unfamiliar code, comparing performance options, updating interfaces, checking deployments, and improving tests. Some conversations start with a stack trace. Others start with a selected function and a short question. The agent is part of the daily engineering work.

For my daily work, I estimate the improvement at at least 10x compared with doing the implementation manually. This is my experience, rather than a controlled benchmark. Counting commits or generated lines would not prove it. The useful result is how much working, reviewed functionality I can complete.

Delegate a Complete Task

With an agent I can delegate a complete, limited task:

"Find how this value reaches the response. Add the new field using the existing pattern, update the relevant callers, and verify the affected behavior."

The agent can search the repository, inspect the code, edit multiple files, and use tools to check its work. I review the result and decide what comes next.

This removes many small interruptions. I spend less time navigating between files and more time thinking about what the application should do.

Start with the Problem

The most expensive mistake is implementing the wrong solution quickly.

One recent investigation started with a service that kept restarting. I asked the agent to inspect the running instance and its memory usage. Then we examined the expensive components, considered alternatives, and worked through the implications before changing the code.

At one point, lazy loading looked like a possible improvement. I asked the agent to check how the loaded data was actually used. If the application eventually reads all the fields anyway, lazy loading may add complexity without removing the expensive work.

We chose a smaller change: process database rows individually instead of holding all the raw rows alongside the decoded objects. Then I asked for tracking so we could understand the remaining cost.

Even there, I challenged the proposed design. Could the existing tracking format hold the information? Did we really need another structure? Would the interface still display it correctly?

These questions are part of the work. The agent makes it much faster to investigate the answers and implement the decision.

I often start with a simple boundary:

"Check only. Do not change anything."

Once I understand the findings, I give the next instruction. This is particularly useful when investigating a running environment or code I do not yet know well.

Stop Repeating the Coding Guidelines

In my earlier post, GO Coding Guideline as Agent Skill, I explained how I converted my coding standards into a reusable skill:

GO Coding Guideline as Agent Skill

Before creating it, I kept asking for the same corrections: follow the naming conventions, reuse existing utilities, simplify the design, use the correct logging, and put the tests in the expected location.

Now these instructions live under .agents/skills/go-guidelines/.

The skill first tells the agent to inspect neighboring code. It then points to separate reference files for style, functions and types, design, errors and logging, runtime safety, and tests. The agent reads the relevant references for the task.

This is useful because the standards include much more than formatting. They cover ownership of state, error handling, configurable limits, lock cleanup, and avoiding secrets in responses or logs.

A skill does not guarantee correct code. However it reduces the amount of repeated explanation and makes the first implementation much closer to what I expect.

I've also added skills for other repeated parts of the workflow.

A Code Review Skill

The code-review skill defines how to review a named branch, from preparing the working tree to producing actionable comments.

It requires a clean working tree and updated branches before reviewing the changes against the development branch. It applies the coding guidelines, checks nearby conventions, and looks for unnecessary complexity.

One particularly useful instruction is checking everything in memory that can grow: caches, collections, buffers, queues, payloads, and concurrent work.

Notice that creating a map with an initial capacity does not limit its growth. A cache with an expiration time can also grow too large if new entries arrive faster than old entries expire. The review should check the actual limits and what happens when they are reached.

The output is deliberately short. Each finding identifies the problem, location, impact, and a concrete fix. If there are no findings, the skill does not create a report file. It also returns to the development branch when finished.

In this workflow, the review skill does not run tests; CI/CD handles them. Implementation checks are a separate part of the task. Keeping this explicit avoids confusing a code review with evidence that the application passed its tests.

A Skill for New Source Files

The add-new-source-files skill handles a small but annoying detail: new source and test files must be staged before the task is finished.

It tells the agent to stage only the source and test files it created, exclude temporary artifacts and unrelated changes, and verify the staged file list. It does not commit unless requested.

Why create a skill for something so simple?

Because a repeated omission is still expensive. A change can look complete in the editor while a required new file is missing from the staged changes. A short instruction makes this part of finishing the task.

A Cloud Credentials Skill

The use-aws-credentials skill records where the agent should load temporary cloud credentials before using the CLI or SDK. It also includes instructions to make the target context and namespace explicit for the relevant infrastructure commands.

The skill points to a local credentials file rather than embedding credential values in its instructions. This saves me from explaining the local setup in every session.

Notice that knowing how to load credentials does not establish that an infrastructure change is appropriate. I still need to understand the target environment and the operation being performed.

The broader idea is simple: write down recurring setup once, then reuse it.

Give the Agent Behavior, Not Just a Feature Name

A useful prompt describes what the user should experience and which behavior must stay consistent.

For example, when asking for multiple views of a graph, the important details include whether each view has its own selected series, whether zoom is shared, and which views can be removed. "Add another graph" leaves all these decisions to the agent.

I give those rules, let the agent implement them, then refine the interaction. Sometimes the next instruction is simply to reuse the existing controls. I do not need a new design language for one feature.

The same applies to backend work. Initializing a component without required data and failing to refresh an already working component are different situations. I ask about both paths. A retry mechanism is only useful if its behavior is correct in each one.

Use the Agent to Build the Comparison

Another useful pattern is asking the agent to create the experiment that will help me decide.

In one conversation, the question was whether an existing cache still justified its complexity. Removing it immediately would have been guessing. We worked on a comparison with a switch that preserved the current default behavior, configurable cases, and a result summary.

I kept asking practical questions: Is there enough coverage to decide? Where does the evidence come from? Can the test use the existing APIs and reduce manual setup? What does a matching result actually mean?

The agent can handle the code needed to make that comparison possible. I still need to decide whether the cases represent real usage and whether the evidence supports removing the cache. Creating a comparison is not the same as proving an optimization worked.

Verify the Running Application

A test passing locally is useful, but sometimes my next question is:

"Is it working inside the running container?"

A dependency change can affect the build, the image, startup, memory usage, and concurrency. I use the agent to check these connections instead of stopping at the updated function.

For a performance-sensitive component, I may ask it to compare alternatives, broaden the sample set, check short and long inputs, and examine behavior with multiple requests. If a test fails after the change, I ask which input failed and what happens next in the application.

A wrong intermediate result matters because of the behavior it triggers downstream. Finding that connection is more useful than changing an assertion until the test passes.

Make Tests Represent Real Behavior

I also use the agent to improve what the tests actually prove.

In one task, I asked for a black-box test that sends several realistic events through the application and examines the exported result. Rather than inventing a convenient input, I asked the agent to use a representative event from local diagnostics as an embedded test resource.

This helps exercise the mapping and processing together. It can expose assumptions that a test of one helper function would miss. Any captured input should be checked and stripped of sensitive values before it becomes a fixture.

For configured limits, I ask for an explicit scenario that reaches the limit and checks what happens next. For a bug, I ask why the existing test did not catch it. Passing tests are more useful when I understand which behavior they cover.

Use the Agent for the Work Around the Code

Development also includes explaining what happened and what someone should do next.

I use the agent to connect interface fields to the data the code actually reads, summarize an authentication failure, and prepare an explanation of the next diagnostic steps. It can turn a long investigation into a short answer grounded in the implementation.

I review these explanations just as I review code. The useful result is an accurate description with clear next steps. Collecting the relevant facts is often the slowest part, and the agent can help with that too.

Keep the Design Simple

Some of my most useful follow-up instructions are short:

"Can we reuse this?"
"This looks duplicated."
"What happens if there is no cached data?"
"KISS. What do you recommend?"

An agent can generate more code very quickly. I want it to help me maintain less code where possible. I also ask it to reuse existing libraries, keep configuration together, and avoid unrelated edits.

When the task grows, these small corrections keep it focused. They also explain why technical experience still matters: I need to recognize when a suggested solution creates more problems than it solves.

Continue in a New Chat Without Starting Over

Long investigations can accumulate many decisions. Before starting a new chat, I ask for a text summary with the information needed to continue.

The useful summary includes the objective, findings, decisions, changes, verification, and remaining questions. It should distinguish code that was edited from code that was actually deployed, and measured results from estimates.

This saves me from explaining the whole investigation again. It also gives the next conversation a clear starting point.

My Daily Agent Workflow

For most development tasks I use the following sequence:

  1. Point to the relevant code, log, screenshot, or running environment.
  2. Ask for an investigation when the cause or design is unclear.
  3. Choose a small implementation step and state the required behavior.
  4. Let the agent update the related code and configuration using the repository skills.
  5. Inspect the changes, question the edge cases, and request relevant verification.
  6. Continue with the next step or save a summary for a new chat.

I use the agent for explanations too. Understanding an unfamiliar subsystem, tracing a configuration value, or finding why a test missed a bug can save as much time as generating code.

"Done" needs evidence appropriate to the task. That might be a passing test, a benchmark comparison, or a verified interaction. If a check could not run, I want that stated clearly.

Where the 10x Gain Comes From

The gain adds up across the entire task.

The agent searches for the existing implementation, follows the pattern across multiple files, updates callers and test support, and handles repetitive edits. The skills reduce repeated instructions and correction rounds. Small tasks keep review manageable.

The savings extend beyond implementation. The same conversation can move from a failure log to the relevant code, from a proposed change to its build implications, and from a test result to the next investigation. I spend less time collecting and moving the information needed for each step.

There is still work for me: deciding whether a cache belongs in the design, whether a new dependency is justified, whether a resource limit is sufficient, and whether the result actually solves the problem.

I discussed the risks in AI Agents Coding Bad Practices:

AI Agents Coding Bad Practices

An agent can make a wrong decision and then create more code to support it. Reviewing small changes helps me catch this before a simple task becomes a complicated subsystem.

Getting Started

Start with one repeated task you understand well. Give the agent a clear expected result and an existing example to follow. Review the changes.

When you find yourself requesting the same correction repeatedly, add it to a focused skill under .agents/skills/. Include when it applies, the steps to follow, and what a completed result looks like.

Then use the saved time to tackle the next task.

My experience is that an AI coding agent becomes far more useful when the repository contains the knowledge needed to work in it. With clear tasks, practical skills, and an experienced engineer reviewing the decisions, I can complete at least 10x more work in my daily development.