ClickHouse backend

Bring your own ClickHouse and unlock the Advanced tier — raw SQL over logs, percentiles, cheap high-cardinality filters, and S3 tiering via storage policies.

The ClickHouse backend writes Rune's log pipeline into a ClickHouse table and lowers Log Explorer queries to SQL. It's the Advanced-tier backend: pick it when you want analytical power over logs, not just retention.

observability:
  enabled: true
  backend: clickhouse
  retention_days: 30                  # DELETE TTL — parts older than this are dropped
  clickhouse:
    dsn: clickhouse://runesight:[email protected]:9000/runesight
    database: runesight               # default
    table: logs                       # default
    auto_migrate: true                # default
    storage_policy: ""                # set to enable S3 tiering (below)
    s3_volume: s3                     # default
    hot_days: 7                       # used with storage_policy

Key	Default	Notes
`clickhouse.dsn`	—	Required. `clickhouse://user:pass@host:9000/db`.
`clickhouse.database`	`runesight`	Created by `auto_migrate` if missing.
`clickhouse.table`	`logs`	Likewise.
`clickhouse.auto_migrate`	`true`	Rune creates the schema on first connect.
`clickhouse.storage_policy`	`""`	Name of a server-side storage policy; setting it enables S3 tiering.
`clickhouse.s3_volume`	`s3`	The policy's volume name parts move to.
`clickhouse.hot_days`	`0`	Parts older than this move to `s3_volume`. `0` = stay hot forever.

Like Loki, this is bring-your-own: ClickHouse Cloud, existing infrastructure, or — since it's an ordinary workload — run it ON Rune with one cast:

git clone https://github.com/runestack/rune-examples
rune cast rune-examples/clickhouse/runeset --release clickhouse \
  --create-namespace --set clickhouse.password=$PASSWORD

Then the DSN host is the in-cluster DNS name: clickhouse.observability.rune:9000.

The schema: auto_migrate vs. operator-managed

With auto_migrate: true (the default), Rune connects lazily on first use and creates the database and table itself — zero-config. The table it creates:

CREATE TABLE IF NOT EXISTS runesight.logs (
  timestamp DateTime64(9) CODEC(Delta(8), ZSTD(1)),
  namespace LowCardinality(String),
  service   LowCardinality(String),
  instance  String,
  node      LowCardinality(String),
  level     LowCardinality(String),
  stream    LowCardinality(String),
  line      String CODEC(ZSTD(1)),
  labels    Map(LowCardinality(String), String),
  INDEX idx_line line TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 4
)
ENGINE = MergeTree
PARTITION BY toDate(timestamp)
ORDER BY (namespace, service, timestamp)

— day partitions, (namespace, service, timestamp) ordering, and a token bloom-filter index on line so |= "needle" filters don't full-scan.

Set auto_migrate: false when a DBA owns the schema: the table must then exist (matching columns) before runed starts ingesting, and Rune will never issue DDL.

Retention and S3 tiering

Both are expressed as table TTLs, applied by auto_migrate:

retention_days: 90 → TTL timestamp + INTERVAL 90 DAY DELETE — parts are dropped wherever they live.
storage_policy + hot_days: 7 → TTL timestamp + INTERVAL 7 DAY TO VOLUME 's3' — parts older than 7 days move from local disk to the policy's S3 volume.

Combined, that's: 7 hot days on NVMe, then S3 until day 90, then gone.

The storage policy itself is server-side ClickHouse configuration — Rune has no S3 client; it only names the policy in the table DDL. The operator adds the policy to ClickHouse's config.d. A complete, copy-paste example:

<!-- /etc/clickhouse-server/config.d/storage.xml -->
<clickhouse>
  <storage_configuration>
    <disks>
      <s3_disk>
        <type>s3</type>
        <endpoint>https://BUCKET.s3.REGION.amazonaws.com/runesight/</endpoint>
        <access_key_id>KEY</access_key_id>
        <secret_access_key>SECRET</secret_access_key>
        <metadata_path>/var/lib/clickhouse/disks/s3_disk/</metadata_path>
      </s3_disk>
    </disks>
    <policies>
      <runesight_tiered>
        <volumes>
          <hot>
            <disk>default</disk>
          </hot>
          <s3>
            <disk>s3_disk</disk>
          </s3>
        </volumes>
        <move_factor>0.1</move_factor>
      </runesight_tiered>
    </policies>
  </storage_configuration>
</clickhouse>

Then in the runefile: storage_policy: runesight_tiered, s3_volume: s3 (the volume name), hot_days: 7. Volume order matters — hot first. Any S3-compatible endpoint works (MinIO, R2, Spaces, GCS interop). The reference runeset wires this XML in via a single value, so tiering works out of the box when you provide bucket + credentials.

Changing storage_policy on an existing table is an ALTER the operator performs; auto_migrate applies the policy at table creation. Plan tiering before first ingest, or be ready to ALTER TABLE ... MODIFY TTL yourself.

What Advanced tier unlocks

The dashboard detects ClickHouse's capabilities and turns on:

SQL mode — a raw SQL editor against the logs table. The escape hatch for anything LogQL can't say: joins against labels, multi-level aggregations, GROUP BY anything.
Percentiles — quantile_over_time(field, 0.95) over parsed numeric fields; powered by ClickHouse's native quantile functions.
High-cardinality filters — filtering on arbitrary labels keys stays cheap at volume (it's a Map column scan with bloom-filter help, not a label-index explosion).

One honest asymmetry: the | logfmt / | json parser stages are not lowered to SQL — Core-tier pipelines that use them run on embedded/Loki but error on ClickHouse. SQL mode is the (more powerful) replacement: JSONExtractString(line, 'field') and friends.

Operational notes

Lazy connection — runed starts even if ClickHouse is down; the forwarder spools and retries until it's reachable.
Sources page — ingestion metrics work as on every backend; store-level disk stats show "managed by the backend" (ClickHouse's system.parts is the authoritative view).
The adapter is integration-tested against a real ClickHouse (testcontainers) in the main repo.