Health checks

A probe tells Rune whether an instance is doing its job. Failed probes mean the reconciler kills and replaces the instance. Get this right.

Probe types

Process-level health (PID alive) is implicit — that's what the runner watches. Probes are for application-level health.

HTTP probe

health:
  liveness:
    type: http
    path: /healthz
    port: 8080
    scheme: http             # or https
    headers:
      X-Health-Check: rune
    initialDelaySeconds: 5
    intervalSeconds: 10
    timeoutSeconds: 2
    failureThreshold: 3
    successThreshold: 1      # must be 1 on liveness; see below

Tunables:

TCP probe

health:
  liveness:
    type: tcp
    port: 5432
    intervalSeconds: 5
    timeoutSeconds: 2

Use TCP when there's no HTTP endpoint to hit — databases, message queues, custom protocols.

Designing a good probe

1. Probe the dependency, not the framework. A /healthz that always returns 200 tells you nothing. A /healthz that pings the DB and returns 503 if it can't is useful.
2. Be cheap. Probes run every 10s × N instances. Don't hit a slow query.
3. Don't probe upstream services. If your probe checks "can I reach the auth API?" then a brief auth blip kills your whole service. Keep probe scope local.
4. Set initialDelaySeconds generously for slow-starting services — JVM, Rails, anything with warm-up.
5. failureThreshold * intervalSeconds is your tolerance window. Default is 30s before kill — usually right.

Inspecting probe state

rune health api --checks
rune health instance api-instance-abc123 --checks

Output includes the most recent probe result, last failure message, and consecutive counts. A failing probe also logs a Liveness check failed / Readiness check failed warning on the server, with the failure under the reason field.

When probes lie

The two failure modes:

• False negatives — your service is fine but probes fail. Cause: tight timeout, expensive endpoint, network blip. The reconciler kills working instances. Fix: relax timeouts, simplify the endpoint.
• False positives — your service is broken but probes pass. Cause: probe doesn't actually exercise the failing path. Fix: have /healthz touch the things that actually matter (DB connection, cache, queue).

Liveness vs. readiness

A service can define two probes under health, and they answer different questions:

• liveness — "is this process still working?" Repeated failures restart the instance (after failureThreshold consecutive failures, default 3, with restart backoff), unless the service sets restartPolicy: never.
• readiness — "is this instance ready to take traffic yet?" It decides when a new instance becomes Running.

health:
  readiness:
    type: http
    path: /ready
    port: 8080
    initialDelaySeconds: 2
    intervalSeconds: 5
  liveness:
    type: http
    path: /healthz
    port: 8080
    initialDelaySeconds: 30

Readiness uses type, path/port, initialDelaySeconds, intervalSeconds and timeoutSeconds as described above, and runs independently of liveness — each probe has its own delay and interval. failureThreshold applies to both probes; successThreshold to readiness only.

What readiness does

• Without a readiness probe, an instance is Running as soon as its container starts. If the app needs 30 seconds to boot, it shows Running and is routed to for those 30 seconds.
• With one, a new instance stays Starting ("Waiting for readiness probe") and receives no traffic until the first readiness probe passes. Then it is promoted to Running and added to the service's endpoints.
• Updates are gated on it. With a readiness probe, a replacement counts as ready when the app is serving. Without one, Rune waits a flat 5 seconds per replacement, and rune lint warns (update-needs-readiness) for services that expose ports, unless the service sets updateStrategy: recreate or is stateful (claimTemplate volumes).
• Dependencies use it. A service waits for each dependency to have at least one Running instance; if the dependency defines a readiness probe, that instance's probe must also be passing.

When readiness fails

Once an instance is Running, a readiness probe that fails failureThreshold times in a row (default 3) withdraws it from traffic. The instance is not restarted and its status stays Running; it is marked not ready, a NotReady event is recorded, and it stays in the service's endpoint set marked unhealthy, so it gets no traffic. After successThreshold consecutive passes (default 1) it is restored and a Ready event is recorded.

• rune get instances shows the state in a READY column: yes, no, or - for an instance that isn't Running. rune get services counts only ready instances in its READY N/M.
• There is no fail-open. If every replica is not ready, the service has no healthy endpoints until one recovers.
• During a rolling update a not-ready instance doesn't count as serving, and the minimum-ready window restarts when it is ready again. A dependency outage therefore can't retire every replica at once: outdated not-ready instances are retired at most one at a time (or as many as the update's allowed dip, set by minServing), paced by replacements becoming ready.
• Instances without a readiness probe are always treated as ready.

Only a failing liveness probe restarts an instance. successThreshold doesn't apply there: one pass clears liveness failures, and a liveness probe that sets successThreshold above 1 is rejected with invalid health check: invalid liveness probe: successThreshold must be 1; it applies to readiness only.

Choosing between them

Point readiness at "can I serve a request right now?" (dependencies reachable, caches warm) and liveness at "is the process wedged?" (a cheap check). A liveness probe that checks external dependencies turns a database outage into a restart loop for every instance.