Atomic File Writes in Go: Prevent Corrupted Files After a Crash

In this post we will review how to save a file in Go without leaving a partially written version when the process crashes. We will use a temporary file and an atomic rename, then explain the additional synchronization needed when the entire machine fails.

This is useful for JSON state files, configuration snapshots, and checkpoints. A file can be small and still be important: if the application cannot read it after restarting, a failed save can become a startup failure.

The examples target Linux and a local filesystem that supports atomic replacement and directory synchronization. The application owns the destination directory, and one writer controls the state being saved.

The Overwrite Problem

Assume our application stores its last processed record in state.json:

{"last_processed_id":100}

After more work, it needs to save 200. The obvious implementation is:

err := os.WriteFile("state.json", data, 0o600)

For an existing file, os.WriteFile truncates it before writing the replacement. If the process stops between these operations, the original content is already gone. If writing fails partway through, the file can contain incomplete JSON.

Checking the returned error is necessary, but it cannot restore the previous file. A sudden process termination also gives our code no opportunity to handle the error.

The Go documentation describes this partial-write behavior.

Reproduce an Interrupted Save

Create a disposable directory and run the examples there. First create a valid state file:

printf '{"last_processed_id":100}\n' > state.json

Save the following program as unsafe.go:

package main

import (
	"log"
	"os"
)

func main() {
	file, err := os.Create("state.json")
	if err != nil {
		log.Fatal(err)
	}
	if _, err := file.WriteString("{\"last_processed_id\":"); err != nil {
		log.Fatal(err)
	}
	os.Exit(1)
}

Run it:

go run unsafe.go
cat state.json

The program deliberately exits with an error after writing only the beginning of the document. The file now contains:

{"last_processed_id":

This is a controlled demonstration of an interrupted overwrite. It does not simulate a power failure, but it makes the application-level problem reproducible without waiting for a real crash.

Notice that defer would not fix this. Deferred functions do not run when the process is killed with SIGKILL or exits through os.Exit.

Prepare the New File Before Replacing the Old One

Instead of changing the current file, prepare its replacement separately:

1. Serialize the complete state in memory.
2. Create a uniquely named temporary file in the destination directory.
3. Write the new content to that file.
4. Synchronize and close it.
5. Rename it over the destination.
6. Synchronize the directory.

Until the rename, state.json remains untouched. On the Linux filesystem assumed here, a reader opening that path sees the old complete file or the new complete file. A reader that already has the old file open can continue reading it.

The temporary file must be on the same filesystem as the destination. Creating it in the destination directory is the simplest way to arrange this. Using the default temporary directory could put it on another filesystem, where the rename fails.

Do not delete the destination before renaming. That would introduce a period where the file is missing and discard the old copy before replacement succeeds.

The Linux rename documentation describes the replacement behavior.

Atomic Replacement and Durability

There are two separate requirements here.

Atomic replacement controls what readers observe while the system is running. They should never open a half-written replacement through the destination path.

Durability concerns what survives a machine crash or power loss. A successful write or close can leave data in operating-system caches. A rename alone does not establish that both the new content and its directory entry reached persistent storage.

The Linux sequence used here is:

write temporary file
sync temporary file
close temporary file
rename temporary file to destination
sync destination directory

The file sync requests persistence of its content and metadata. The directory sync requests persistence of the name change. Both operations can fail, and their errors must be handled.

This is the standard persistence sequence for this Linux use case, subject to the filesystem and storage stack honoring these operations. It does not protect against failing hardware or replace backups.

The Linux fsync documentation explains why the directory requires its own synchronization.

Go Implementation

The following program uses only the standard library and requires Go 1.20 or later because it uses errors.Join. Save it as main.go and run it separately from unsafe.go:

package main

import (
	"errors"
	"fmt"
	"log"
	"os"
	"path/filepath"
)

func writeFileAtomic(
	filePath string,
	data []byte,
) error {
	directory := filepath.Dir(filePath)
	temporary, err := os.CreateTemp(directory, ".state-*.tmp")
	if err != nil {
		return fmt.Errorf("create temporary file: %w", err)
	}
	defer temporary.Close()
	defer func() {
		err := os.Remove(temporary.Name())
		if err != nil && !errors.Is(err, os.ErrNotExist) {
			log.Printf("remove temporary file: %v", err)
		}
	}()

	if _, err := temporary.Write(data); err != nil {
		return fmt.Errorf("write temporary file: %w", err)
	}
	if err := temporary.Sync(); err != nil {
		return fmt.Errorf("sync temporary file: %w", err)
	}
	if err := temporary.Close(); err != nil {
		return fmt.Errorf("close temporary file: %w", err)
	}
	if err := os.Rename(temporary.Name(), filePath); err != nil {
		return fmt.Errorf("replace destination: %w", err)
	}
	if err := syncDirectory(directory); err != nil {
		return fmt.Errorf("destination replaced, directory sync failed: %w", err)
	}
	return nil
}

func syncDirectory(
	directory string,
) error {
	file, err := os.Open(directory)
	if err != nil {
		return err
	}
	syncError := file.Sync()
	closeError := file.Close()
	return errors.Join(syncError, closeError)
}

func main() {
	data := []byte("{\"last_processed_id\":200}\n")
	if err := writeFileAtomic("state.json", data); err != nil {
		log.Fatal(err)
	}
	fmt.Println("Saved state.json")
}

Run the safe version:

go run main.go
cat state.json

The destination now contains:

{"last_processed_id":200}

os.CreateTemp creates a unique file with restrictive permissions, 0o600 before applying the process umask. This example retains those permissions when the file becomes state.json.

The deferred close is a fallback for early failures. The explicit close before rename is checked. Deferred removal cleans up an unpublished temporary file during a normal return; after a successful rename, the temporary pathname no longer exists.

The helper expects the parent directory to exist and remain stable during the operation. Create the application's state directory during initialization. Persisting a newly created directory hierarchy requires additional synchronization of its parent directories.

What Happens at Each Failure Point?

If the temporary write, file sync, or close fails, the rename does not happen. An existing destination remains unchanged.

If the process stops before rename, an existing destination still contains the old version. A temporary file may remain because deferred cleanup did not run.

If rename succeeds, new opens of the destination see the replacement. If the process then stops before directory synchronization, its persistence across a later machine failure has not been established.

If directory synchronization fails, the function returns an error that explicitly says the destination was already replaced. We cannot treat every error as meaning that nothing changed. The new version is visible, but its durability was not confirmed.

On the first save there is no old version to preserve. A process interruption before rename leaves the destination absent, possibly alongside an unfinished temporary file.

These distinctions matter when deciding whether to retry or report success. A checkpoint should also be coordinated with the work it represents; safely writing the file does not make an external side effect and the checkpoint one transaction.

The same uncertainty appears when a connection breaks after a server has performed its work, as discussed in HTTP Streaming Error Handling in GO: Why HTTP 200 Is Not Enough.

Verify the Failure Paths

Reset state.json to 100. In a disposable copy of main.go, insert os.Exit(1) immediately before os.Rename and run it again.

state.json should still contain 100. The completed temporary file should remain beside it. Repeat with an exit after writing only part of the temporary content: the destination should still be unchanged.

Remove the injected exit and run the normal version. The destination should contain 200, and the operation should leave no temporary file behind.

Also check a failed rename and a failed directory sync. The first should preserve the existing destination. The second should report an error while leaving the new content visible.

These checks verify the process-interruption behavior. They do not prove recovery after a power loss; that requires testing the actual filesystem and storage environment with an appropriate crash-testing setup.

Permissions and Concurrent Writers

Replacing a file installs a new filesystem object. It does not automatically preserve the old file's permissions, ownership, ACLs, or extended attributes. If specific metadata is required, apply it to the temporary file before its sync and publication.

The example also assumes the destination is a regular state-file path, not a symbolic link whose target must be updated. Renaming onto a symbolic link replaces that link itself.

Unique temporary names prevent writers from sharing the same temporary file. They do not prevent lost updates.

Assume two workers read 100, independently calculate 101, and each publish a complete file. Both files are valid, but one update is lost. Within one process, serialize the entire read-modify-write operation. Across processes, use appropriate coordination or a transactional database.

This is related to the distinction between individual atomic operations and a complete synchronized operation discussed in Go Atomic Value is NOT sync.

Operational Limits

A replacement temporarily needs space for both versions. Near a disk limit, even a small state update can fail during writing or synchronization. Keep the old version and surface the failure instead of continuing as if the state was saved.

Abrupt exits can leave temporary files. If startup cleanup removes them, ensure no active writer can still own them; a broad filename match alone is not sufficient.

Go does not promise atomic os.Rename behavior on every operating system. Windows, network filesystems, container volume drivers, and object-storage mounts require checking their own replacement and persistence semantics. Do not infer those guarantees from a successful test on a local Linux directory.

For several files that must change together, atomic replacement of each file separately is insufficient. Use a transactional store or a publication scheme that switches one reference to a complete version.

When the state already belongs in PostgreSQL, a database transaction can also keep a business update and its queued work together, as shown in Build a PostgreSQL Job Queue with SKIP LOCKED.

Final Note

For a small state file, the useful pattern is straightforward: prepare the complete replacement, sync and close it, publish it with a same-filesystem rename, then sync the directory when durability is required.

This keeps an interrupted save from destroying the last complete version and makes the remaining boundaries explicit: persistence, concurrent updates, and recovery after an uncertain result.