Skip to content

Backup: the operator runs backups and restores itself through SQL on clickhouse-backup sidecars, multi-period schedules - #2148

Open
Slach wants to merge 1 commit into
0.28.0from
backup-config-file
Open

Slach wants to merge 1 commit into
0.28.0from
backup-config-file

Conversation

@Slach

@Slach Slach commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

fix #2146

What changes

The operator itself runs backups and restores. There is no orchestrator script, no Job, no CronJob and no backup Pod: the operator drives the clickhouse-backup sidecar of every host through SQL on the integration tables of that host's ClickHouse, as the Altinity usual backup workflow does.

  • Transport: sidecar only. Every Pod of a backup-enabled cluster gets a clickhouse-backup server sidecar. Its configuration, generated by the operator, always turns on api.create_integration_tables, so system.backup_actions and system.backup_list read http://127.0.0.1:7171. The sidecar runs with TZ=UTC, and restore_distributed_cluster defaults to {cluster}. backup.mode and pod mode are removed.

  • Credentials. The operator connects to each host by its FQDN, port, scheme and the cluster TLS, with the ClickHouse credentials of its own configuration (clickhouse.access or its Secret). They must be set explicitly: the built-in defaults do not count, and without them a backup or restore waits with reason NoClickHouseAccess.

  • Commands. A command is a row inserted into system.backup_actions of the host after now('UTC') of that host (async_insert = 0, http_receive_timeout/http_send_timeout from httpReceiveTimeout), and polled every 5 seconds, matched by its exact command text and start time, until it finishes. A busy sidecar (another operation is currently running) is retried with exponential backoff from 5 seconds up to 5 minutes. After an operator restart the leader resumes every run from its status: a running command is followed, a finished one is settled.

  • Timeouts. spec.timeouts of a ClickHouseBackup, ClickHouseRestore and ClickHouseBackupSchedule, defaults in the operator configuration backup.timeouts and in code, as Go durations: actionTimeout (12h), unreachableTimeout (10m), httpReceiveTimeout (4h), errorResponsesMax (5). Errors and empty polls are counted apart; the logs carry the response code.

  • Backup. create_remote --rbac on one replica per shard (replicaSelection: FirstPerShard, preferredReplica, fallback to another replica reported by OrchestratorReplicaFallback) or on every replica (AllReplicas). A shard is backed up once any of its replicas made the backup. A failed backup deletes only its local copy, once, and reports a remote leftover (OrchestratorFailedBackupLeft, OrchestratorFailedBackupRemoteLeft) with the SQL to delete it by hand.

  • Storage paths. Before any command, path and object_disks_path of every writer of a backup, with the macros of each host expanded, must be unique (AllReplicas: every replica; embedded backups and CAS disks included). Otherwise the backup or schedule is Aborted with StoragePathConflict, e.g. storage path 's3://backups/backup/main/0' is not unique: replicas ... both write a backup of one name there. Unknown macros give an OrchestratorStoragePathAdvice warning.

  • Backup names. spec.backup.naming / clusters[].backup.naming, a template such as shard{shardNum}-{YYYYMMDDHHIISS} with {shard} (the shard macro), {shardNum} (shard_num, from 1), {cluster}, {chi}, {namespace}, {prefix} and the time tokens. Retention, the incremental base and restore parse names back with it, so backups named by the Altinity usual backup workflow on a shared remote path are recognized. Empty keeps the current names.

  • Schedules. ClickHouseBackupSchedule.spec.schedules[] (minute, hour, 6hours, daily, weekly, monthly, with time, day, keep, incremental, configOverrides) and spec.timeZone:

    • the operator runs the slots of the finest entry itself; the heaviest matching entry decides full vs incremental;
    • no CronJob, Job or ClickHouseBackup is created per run: a run lives in status.active while it runs and in status.recentRuns afterwards, bounded by successfulRunsHistoryLimit (3) and failedRunsHistoryLimit (1); startingDeadlineSeconds skips a slot missed by more; runs of one cluster never overlap;
    • a ClickHouseBackup with scheduleRef runs the schedule now: its incremental entry, retention after it, a base of the next incremental backup.
  • Retention follows the Altinity usual backup workflow: each entry keeps its own keep newest backups (GFS); the newest backup is always kept; run-now backups live within the window of the scheduled ones; an incomplete backup is deleted only once a newer complete one exists and system.backup_actions shows no create_remote or upload of it on its shard; nothing is deleted during a restore; incremental bases are rebased by clickhouse-backup (forced backups_to_keep_remote: 0, rebase_*: true).

  • Schedule changes. spec.retention.onScheduleChange: Hold (default) records a change of the entries or the time zone in status.retention.previous[] with its expires. The backups the schedule kept before it and the current one would delete are held, listed in status.clusters[].heldBackups with expires: backup time + keep × the period of the longest old entry that kept it. Expired holds are deleted (OrchestratorRetentionHeldExpired), an expired change leaves previous[] (RetentionHeldExpired).

    • keepBackups: [{name, until}] pins backups: with until until then; without it a backup held by a change until its expiry, any other backup while it is listed. Pins protect incomplete backups too.
    • releaseHeld: <changeID> releases a change and all earlier ones, Prune applies a change at once.
  • Config overrides. schedules[].configOverrides and ClickHouseBackup.spec.configOverrides: up to 16 KEY=VALUE clickhouse-backup settings for one backup, sent as create_remote --env=.... Keys the operator pins (TZ included), API_*, paths, buckets, endpoints, the embedded backup disk and secrets are refused.

  • Restore, as in the Altinity usual backup workflow:

    1. download --hardlink-exists-files on every host (an incomplete local copy is resumed);
    2. restore --schema=1 --ignore-dependencies on every replica, the first replica of the first shard with RESTORE_SCHEMA_ON_CLUSTER={cluster}; skipped for partitions;
    3. spec.replicatedCopyToDetached (default true): the other replicas of each shard run restore --data=1 --ignore-dependencies --replicated-copy-to-detached at the same time, so the parts of replicated tables are already local; with false they download only the schema;
    4. restore --data=1 --ignore-dependencies on one replica per shard, the next answering replica when the first does not;
    5. delete local on every host after success; a failed restore keeps its downloads.

    Filters go on every command: tables, partitions, skipTables, skipTableEngines, databaseMapping, rbac. overwrite: true without tables adds --rm, which recreates the databases of the backup as a whole; with tables, or without overwrite, only the tables of the backup are recreated (clickhouse-backup drops the existing ones) and no database is dropped. A restore of access entities holds the backups and schedules of its own clusters only; restores of different clusters run at the same time.

  • Reruns: spec.taskID on all three kinds, with status.taskID, taskIDsStarted, taskIDsCompleted. A completed backup runs again under a new name (the old one in status.previousBackupNames), an unfinished one under its own name after its earlier copies are deleted. A failed restore runs again: spec.resumeDownload (default true) resumes its earlier downloads, false deletes them and starts over. An aborted schedule is validated again. A taskID set while a run is active is ignored with a TaskIDIgnored warning.

  • Deletion. Finalizers on ClickHouseBackup, ClickHouseRestore and ClickHouseBackupSchedule. Deleting an unfinished backup kills only its own running commands (kill "<command>") and deletes its local and remote copies; a completed one keeps its remote backup for retention. A restore and a schedule run are waited for, and a deleted schedule starts no new slot. Unreachable hosts are given up after unreachableTimeout, with a DeletionLeftover warning and the SQL to finish by hand.

  • Validation. Backup enabled without configSecretRef, a missing or invalid storage Secret, or an invalid backup block stops the CHI reconcile: status Aborted, reason BackupConfigInvalid, Warning event BackupStorageConfigInvalid.

  • Status and events. status.clusters[].run of a ClickHouseBackup / ClickHouseRestore records the step, the hosts and every command sent with its status. Phases Pending, Running, Completed, Failed, Aborted. The operator posts every event itself (Orchestrator* per step and command) on the ClickHouseBackup, ClickHouseBackupSchedule and ClickHouseRestore.

  • Removed: backup.mode and pod mode, backup.orchestrator, the bash and Python orchestrator scripts and their ConfigMaps, orchestratorConfigMapRef, backup.path of the operator configuration, jobName / cronJobName, concurrencyPolicy, backup verification (spec.verify, the Verified condition and its metric), the restore overwrite guard, and the operator RBAC rights for Jobs, CronJobs, Pod writes, ServiceAccounts, Roles and RoleBindings. The operator gains update/patch on its three backup kinds for the finalizers.

  • Manifests are regenerated with the current yq, which writes plain <<: merge keys in place of !!merge <<:; the documents are the same.

  • Docs: docs/backup.md (how the operator runs an operation, naming, timeouts, storage paths, migration from the Altinity usual backup workflow, schedule changes, reruns, deletion, config overrides, integration tables), docs/custom_resource_explained.md, docs/operator_configuration.md, docs/chb-examples/.

Tests

  • go build ./... && go test ./..., -race for pkg/controller/chbackup and pkg/model/chbackup: runs against fake sidecars (command text, polling, timeouts, resume after restart, busy backoff, replica fallback, copy to detached, finalizers, taskID reruns, path uniqueness, per-cluster gates), retention plans, schedule changes and their expiry, naming, slots, incremental base.
  • helm template of the regenerated chart.
  • Native e2e on orbstack:
    • test_010087: backup, schedule, retention and restore of two clusters through real sidecars;
    • test_010088: recovery of a new CHI from another one's backup;
    • test_010089: retention across a schedule change: held backups expiring by the old schedule, keepBackups pins without until, with until in the past and in the future.

Related, in clickhouse-backup:

deploy/operatorhub is not regenerated (done at release). dev/run_code_generator.sh does not build with the current Go (#2152); deepcopy was generated with deepcopy-gen v0.34.1.

🤖 Generated with Claude Code

@Slach
Slach force-pushed the backup-config-file branch from d0d0ec5 to d81d518 Compare November 12, 2026 10:51
@Slach Slach changed the title Backup: sidecar or per-shard Pod transport, multi-period schedules, orchestrator script as operator config Backup: sidecar or per-shard Pod transport, multi-period schedules, bash or Python orchestrator as operator config Nov 13, 2026
@Slach
Slach force-pushed the backup-config-file branch from ddc3980 to c3e4b29 Compare November 16, 2026 14:54
@Slach Slach changed the title Backup: sidecar or per-shard Pod transport, multi-period schedules, bash or Python orchestrator as operator config Backup: the operator runs backups and restores itself through SQL on clickhouse-backup sidecars, multi-period schedules Nov 17, 2026
@Slach
Slach force-pushed the backup-config-file branch from c3e4b29 to 3c76ca0 Compare November 17, 2026 13:26
… the sidecars

Operator-managed backup no longer creates Jobs, CronJobs or backup Pods and ships
no orchestrator script. The operator drives the clickhouse-backup sidecar of every
host through the integration tables of its ClickHouse - system.backup_actions and
system.backup_list - with the ClickHouse credentials of its own configuration, one
run per cluster, as the Altinity usual backup workflow does.

- Runs: a command is a row of system.backup_actions, inserted after now('UTC') of
  the host and polled every 5 seconds until it finishes; a lost command is settled
  (a resumed upload or download is followed, a complete backup counts as made); a
  busy sidecar is retried with exponential backoff from 5 seconds to 5 minutes.
- Status: status.clusters[].run of a backup or a restore, and status.active and
  status.recentRuns of a schedule, record each run - step, hosts, the commands sent
  with their status, the plan - and the leader resumes runs after a restart.
- Schedules run at the slots of their finest entry without a CronJob or a
  ClickHouseBackup per run; startingDeadlineSeconds skips a slot missed by more;
  successful/failedRunsHistoryLimit bound status.recentRuns.
- Credentials: backup runs only with a ClickHouse username and password the operator
  configuration sets itself, in clickhouse.access or its Secret; the defaults do not
  count (reason NoClickHouseAccess).
- A shard is backed up once any of its replicas made the backup, AllReplicas
  included; retention takes a backup as complete per shard.
- Restore follows the Altinity usual backup workflow: download
  --hardlink-exists-files on every host, restore --schema=1 on every replica - the
  first replica of the first shard with RESTORE_SCHEMA_ON_CLUSTER={cluster} -,
  restore --data=1 on one replica per shard, then delete local. Every restore
  command takes --ignore-dependencies; --rm, which recreates the databases of the
  backup whole, only for an overwrite without tables; the filters (tables,
  partitions, skipTables, skipTableEngines, databaseMapping, rbac) go on every
  command. The
  overwrite guard, the separate ON CLUSTER drop pass, the max_table_size_to_drop
  checks and the dependency on the ZooKeeper configuration of the CHI are gone.
- The sidecar pins API_CREATE_INTEGRATION_TABLES and API_INTEGRATION_TABLES_HOST;
  the operator reaches each host by its FQDN, port, scheme and cluster TLS.
- Events of the runs (Orchestrator*), conditions and reasons (Started, Failed,
  FailedUnchanged, NoClickHouseAccess) are posted by the operator.
- Removed: backup.mode and pod mode, backup.orchestrator, the bash and Python
  orchestrator scripts and their ConfigMaps, orchestratorConfigMapRef, jobName and
  cronJobName, concurrencyPolicy, backup verification (spec.verify, the Verified
  condition and its reasons, the verification metric), backup.path of the operator
  configuration, and the RBAC rights for Jobs, CronJobs, Pods and RBAC objects.
- Names: {shard} of a naming template is the shard macro, {shardNum} the shard_num
  from 1; restore keeps the naming prefix.
- The sidecar runs with TZ=UTC; restore_distributed_cluster defaults to {cluster}.
- spec.timeouts of backups, restores and schedules, with defaults in the operator
  configuration (backup.timeouts) and in code: actionTimeout (12h),
  unreachableTimeout (10m), httpReceiveTimeout (4h), errorResponsesMax (5), as Go
  durations. Errors and empty polls are counted apart, with the response code.
- A failed backup deletes only its local copy, once; a failed restore keeps its
  downloads, deleted only after success.
- Retention: keepBackups pins {name, until}; backups held by a schedule change, and
  pins without until of such backups, expire by the schedule before the change
  (backup time + keep x period of its longest entry that kept it), listed in
  heldBackups with expires; an incomplete backup is deleted only when no
  create_remote or upload of it runs on its shard per system.backup_actions.
- Storage paths: path and object_disks_path, macros expanded per host, must be
  unique among the writers of one backup (AllReplicas: every replica), embedded
  backups included; otherwise the backup or schedule is Aborted with
  StoragePathConflict before any command.
- Finalizers: deleting an unfinished backup kills its commands and deletes its
  local and remote copies; a completed one keeps its remote backup; a restore and a
  schedule run are waited for.
- spec.taskID: a completed backup runs again under a new name, an unfinished one
  under its own; a failed restore runs again, resuming its downloads
  (spec.resumeDownload, default true) or starting over; an aborted schedule is
  validated again. A taskID during a run is ignored with a warning.
- Restore: spec.replicatedCopyToDetached (default true) - the other replicas of a
  shard restore the data of replicated tables to detached first, then one replica
  attaches it; false downloads only the schema on them.
- A restore of access entities holds the backups and schedules of its own clusters
  only; restores of different clusters run at the same time.
- Manifests regenerated with the current yq: plain <<: merge keys.
- CRDs, manifests, Helm chart, docs/backup.md and the examples follow; unit tests
  cover the runs on fake sidecars; e2e test_010087, test_010088 and test_010089
  (retention across a schedule change, keepBackups) cover real sidecars.
- ci: actions/setup-python@v7, which runs on Node.js 24.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant