Volume backups with restic

Provision a bucket and S3 keys at your storage provider

One-time, outside yoink. Any S3-compatible backend works; for a self-hosted RustFS path see Self-hosted backend below.

Render the backup service from the bundled template

yoink add restic-backups \
  --var "repository=s3:https://s3.example.com/MY-BUCKET" \
  --var "schedule=0 30 3 * * *" \
  --var "retention=--keep-daily 7 --keep-weekly 4 --keep-monthly 6"

Seal the operator-supplied S3 keys

RESTIC_PASSWORD was auto-generated by the template; only the S3 keys are operator-supplied.

yoink secrets edit
#    S3_ACCESS_KEY_ID=… [step]
#    S3_SECRET_ACCESS_KEY=… [step]

List which volumes to back up, then deploy

Edit services/backups.yaml to uncomment + repoint the example bind mounts, then:

yoink up

After the next cron tick (default 0 30 3 * * * = 03:30 UTC), restic snapshots against the bucket lists your first backup.

Deploy RustFS on the off-site host

The host must already be in your hosts: list and reachable over the tailnet.

yoink add rustfs --var "bind_address=100.64.1.5"   # the host's tailnet IP

Edit services/rustfs.yaml to pin hosts: [vault-host.tailnet] so it only deploys on the off-site box, then yoink up.

Pre-create the data volume with the right ownership

RustFS runs as uid 10001; chown its data volume so the daemon can write to it before the first deploy.

ssh root@<vault-host> '
  docker volume create rustfs-rustfs-data
  chown -R 10001:10001 /var/lib/docker/volumes/rustfs-rustfs-data/_data
'

Create the backup bucket

restic doesn't auto-create S3 buckets, only the repo metadata inside one. After RustFS is up, use any S3 client against the running RustFS:

ssh root@<vault-host> "docker run --rm --network=<your-network> \
  -e AWS_ACCESS_KEY_ID='$(yoink secrets show RUSTFS_ACCESS_KEY --reveal | cut -d= -f2-)' \
  -e AWS_SECRET_ACCESS_KEY='$(yoink secrets show RUSTFS_SECRET_KEY --reveal | cut -d= -f2-)' \
  -e AWS_REGION=us-east-1 \
  amazon/aws-cli --endpoint-url=http://rustfs:9000 s3 mb s3://backups"

Render the backups service against the RustFS endpoint

yoink add restic-backups \
  --var "repository=s3:http://rustfs-vault.tailnet:9000/backups"

Reuse RustFS credentials as the S3 keys

The auto-generated RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY are the S3 root credentials for that RustFS instance. Copy them into the S3_ACCESS_KEY_ID / S3_SECRET_ACCESS_KEY slots that restic-backups expects:

yoink secrets edit
#  S3_ACCESS_KEY_ID    <- paste from `yoink secrets show RUSTFS_ACCESS_KEY --reveal` [step]
#  S3_SECRET_ACCESS_KEY <- paste from `yoink secrets show RUSTFS_SECRET_KEY --reveal` [step]

Deploy and watch dedup do its thing

yoink up deploys the backups service. Watch the second backup land; only changed chunks ship. See pairing with Tailscale for tailnet hostname conventions.

Make a single bucket world-readable

Object-level access only, not bucket-listing. Run from any host on the same docker network.

POLICY='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::public-assets/*"}]}'
docker run --rm --network=<your-network> \
  -e AWS_ACCESS_KEY_ID="$(yoink secrets show RUSTFS_ACCESS_KEY --reveal | cut -d= -f2-)" \
  -e AWS_SECRET_ACCESS_KEY="$(yoink secrets show RUSTFS_SECRET_KEY --reveal | cut -d= -f2-)" \
  -e AWS_REGION=us-east-1 \
  amazon/aws-cli --endpoint-url=http://rustfs:9000 s3api put-bucket-policy \
    --bucket public-assets --policy "$POLICY"

Add a domain: and Caddy rewrite to services/rustfs.yaml

The rewrite injects the bucket prefix so /foo.png at the public domain maps to /public-assets/foo.png against RustFS; yoink's auto-generated reverse_proxy rustfs:9000 (emitted because domain: is set) handles the actual proxying with the rewritten path.

domain: assets.example.com
caddy_extra_caddyfile: |
  rewrite * /public-assets{path}

Deploy and hit the public domain

After yoink up, https://assets.example.com/foo.png serves the object from the public-assets bucket. TLS is via Let's Encrypt by default (or sealed origin certs; see Cloudflare origin certs). The path-rewrite scopes the public endpoint to one bucket so the rest of your S3 endpoint (other buckets, the admin API) stays internal-only. Use this only for genuinely public assets, since the bucket policy is the sole access control on objects under public-assets/*. Add a method matcher (@get method GET HEAD + respond 405 for everything else) if you want defense-in-depth against rogue PUT attempts.

Provision a WAL bucket

The bucket must exist before postgres starts archiving; restic-style auto-init does not apply to archive_command.

Seal the S3 keys (or reuse the ones from restic-backups)

WAL upload is just aws s3 cp, so the same keys cover both flows.

yoink secrets edit
#   S3_ACCESS_KEY_ID=… [step]
#   S3_SECRET_ACCESS_KEY=… [step]

Render postgres with WAL archiving on

yoink add postgres \
  --var wal_archiving=true \
  --var wal_repository=s3://wal-archive \
  --var wal_endpoint=http://rustfs:9000   # omit for AWS S3

Pair with restic-backups against the data volume

Bind the postgres data volume read-only into the backups service so the base snapshot half of PITR is in place too.

# services/backups.yaml [step]
- pgdata-postgres:/data/pgdata-postgres:ro

This emits a service that wraps the stock postgres entrypoint with a one-line apk add aws-cli and starts postgres with archive_mode=on, wal_compression=zstd (full-page images compressed inside WAL records), and an archive_command that pipes each segment through gzip -c before upload, so the on-disk WAL is denser before it ships and the segment files in S3 are gzip-compressed .gz blobs (idle-database segments compress from 16 MiB to a few KiB; busy ones to ~3-6 MiB). archive_timeout=300 forces a flush every 5 min so an idle database still ships its tail.

Restore the latest pre-incident base snapshot

restic restore the latest pre-incident snapshot of pgdata-postgres to a fresh volume.

Wire restore_command to the WAL archive

Add a recovery.signal file plus restore_command to the recovered data dir. The .gz suffix and gunzip mirror the archive-side compression:

restore_command = 'aws s3 cp s3://wal-archive/%f.gz - | gunzip -c > %p'

Set the recovery target and start postgres

Set recovery_target_time to the desired moment, start postgres, wait for replay to complete.

Full PITR mechanics live in the PostgreSQL continuous archiving docs. The yoink piece is just the upload half; archive_command is the only contract postgres has with the storage backend.

WAL archive retention is currently a manual concern: aws s3 rm (or RustFS lifecycle rules) older than your restic retention window. A future iteration of this recipe may bundle pg_archivecleanup as a sidecar; for now keep WAL retention ≥ your oldest restic snapshot.

Tail go-cron's schedule log

yoink up
yoink logs backups -f

You should see go-cron logging the next scheduled run. Force a snapshot immediately by uncommenting RUN_ON_STARTUP: "true" in services/backups.yaml, redeploying, then flipping it back off once you've confirmed.

List snapshots from your laptop

With the same secrets in env, list snapshots against the same repo URL:

docker run --rm \
  -e RESTIC_REPOSITORY="s3:https://s3.example.com/MY-BUCKET" \
  -e RESTIC_PASSWORD="$(yoink secrets show RESTIC_PASSWORD --reveal)" \
  -e AWS_ACCESS_KEY_ID="$(yoink secrets show S3_ACCESS_KEY_ID --reveal)" \
  -e AWS_SECRET_ACCESS_KEY="$(yoink secrets show S3_SECRET_ACCESS_KEY --reveal)" \
  mazzolino/restic:1.8.2 \
  restic snapshots