Instances

An instance is a single running copy of a service. If you have scale: 3, you have 3 instances. They're created and destroyed by the reconciler — you never create one directly.

Identity

Each instance has:

• A stable ID like api-instance-7c2e8a3b.
• A reference back to its service and generation.
• A runner type (docker or process).
• A state (Pending, Starting, Running, Unhealthy, Stopping, Stopped, Failed).
• The node it's running on (single-node today; multi-node tomorrow).

rune get instances
rune get instance api-instance-7c2e8a3b -o yaml

Lifecycle

Pending → Starting → Running ──┬──▶ Unhealthy → Stopping → Stopped
                               │
                               └──▶ Stopping → Stopped
                                                   │
                                                   └─▶ (replaced if scale unchanged)

The reconciler will replace failing instances automatically. Permanent failures (e.g., image pull errors that don't resolve) bubble up to the service status.

Stalled — stuck in create

When create fails repeatedly after repeated create failures — an unpullable image, a rejected registry credential, a missing secret or storage class — the reconciler exhausts its retries and freezes the record in Stalled. It stops auto-retrying on purpose: a service that cannot pull its image would otherwise burn instance IDs forever. The slot stays held by that record, so no replacement appears either.

A Stalled instance never reached a running container, so rune logs and rune exec have nothing to attach to. Read the cause off the service:

rune describe service api -n default

Fix the cause, then re-arm the slot:

rune restart api

rune restart clears the create-attempt counter and the retry backoff for these records, so the reconciler tries again immediately. See Recovering instances stuck in create.

Operations on instances

Killing an instance returns it to the reconciler, which replaces it. Don't use that as "scale up" — use rune scale.

Instance vs. service in commands

Most commands accept either a service name or an instance ID. Some, like exec, attach to a specific instance — if you pass a service name, Rune picks one for you.

rune exec api bash # picks any healthy instance of 'api'
rune exec api-instance-7c2e bash # specific instance

Per-instance state

Each instance carries:

• Conditions[] — point-in-time check results (Ready, Healthy, ImagePulled, etc.).
• Restarts — how many times the runner has restarted it.
• LastFailureMessage — last error from the runner.
• StartedAt / FinishedAt.

Use rune get instance <id> -o yaml to read all of it. The most useful field for debugging is LastFailureMessage.

Postmortem: failed-instance retention

When an instance is restarted because of a health-check failure, Rune preserves the failing container as a tombstone instead of removing it straight away. The tombstone keeps the underlying docker container in its stopped state so operators have time to inspect it — docker logs, the container filesystem, the env it crashed with — instead of racing the reconciler.

Tombstones:

• Appear in rune get instances --show-failed (hidden by default).
• Have Status=Failed and a populated FailedAt / FailureReason.
• Are named <original-name>-failed-<short-uuid>.
• Keep working with rune logs <tombstone-id> until they're reaped.
• Don't count against the service's desired scale — the reconciler treats them as bookkeeping, not live instances.

# Find postmortem candidates for a flapping service
rune get instances -n prod --show-failed
 
# Inspect the dead container's logs
rune logs landing-0-failed-a3b9c1d8 -n prod
 
# Postmortem exec (the failing app does NOT re-run — server starts an
# inspection container with a sleep entrypoint instead).
rune exec --debug landing-0-failed-a3b9c1d8 -- ls /etc/secrets

Retention policy

A retention GC pass on the reconciler bounds how many tombstones survive and for how long. Defaults are baked into the orchestrator for v1 (per-service cap of 3 tombstones; TTL of 1h, whichever fires first); a failed_instance_retention block in the runefile lets operators override.

# runefile
failed_instance_retention:
  per_service_cap: 5       # keep up to 5 most-recent tombstones per service
  ttl: 2h                  # hard-evict anything older than 2h
  snapshot_log_bytes: 200000  # reserved for v2 (post-eviction log snapshot)

When a tombstone is evicted, its container is removed and the instance record is marked Deleted; the existing deleted-instance retention sweep cleans the store row a few minutes later.

How --debug works under the hood

When you run rune exec --debug <tombstone-id> -- bash, the server:

1. Looks up the tombstone (must be Status=Failed with a preserved ContainerID — rejects Running instances with "drop the flag" and already-evicted tombstones with a "use rune logs --previous" hint when that lands).
2. Builds a fresh docker container config from the tombstone's image, env, mounts, volumes, resources — same template the original used.
3. Replaces the entrypoint with sleep infinity so the failing app does not re-run when the new container starts. This is the whole reason a flag exists instead of just docker start on the stopped container.
4. Names the sidecar <tombstone-name>-debug-<short-uuid> and tags it with a rune.debug=true label.
5. Pulls the image with IfNotPresent policy (usually a no-op since the failing instance just ran it).
6. Starts the sidecar, opens an exec session against it for your command.
7. Tears down the sidecar (stop + remove) when your session ends — clean exit, Ctrl+C, or transport disconnect all trigger cleanup via a sync.Once-guarded teardown wrapper on the exec stream.

The original failed container is never touched, so rune logs <tombstone-id> still works on it afterwards. Only the docker runner supports --debug today; the process runner returns FailedPrecondition because there's no image to clone.