Repository navigation
Conversation
Slach
force-pushed
the
backup-config-file
branch
from
November 12, 2026 10:51
d0d0ec5 to
d81d518
Compare
Slach
force-pushed
the
backup-config-file
branch
from
November 16, 2026 14:54
ddc3980 to
c3e4b29
Compare
Slach
force-pushed
the
backup-config-file
branch
from
November 17, 2026 13:26
c3e4b29 to
3c76ca0
Compare
… the sidecars
Operator-managed backup no longer creates Jobs, CronJobs or backup Pods and ships
no orchestrator script. The operator drives the clickhouse-backup sidecar of every
host through the integration tables of its ClickHouse - system.backup_actions and
system.backup_list - with the ClickHouse credentials of its own configuration, one
run per cluster, as the Altinity usual backup workflow does.
- Runs: a command is a row of system.backup_actions, inserted after now('UTC') of
the host and polled every 5 seconds until it finishes; a lost command is settled
(a resumed upload or download is followed, a complete backup counts as made); a
busy sidecar is retried with exponential backoff from 5 seconds to 5 minutes.
- Status: status.clusters[].run of a backup or a restore, and status.active and
status.recentRuns of a schedule, record each run - step, hosts, the commands sent
with their status, the plan - and the leader resumes runs after a restart.
- Schedules run at the slots of their finest entry without a CronJob or a
ClickHouseBackup per run; startingDeadlineSeconds skips a slot missed by more;
successful/failedRunsHistoryLimit bound status.recentRuns.
- Credentials: backup runs only with a ClickHouse username and password the operator
configuration sets itself, in clickhouse.access or its Secret; the defaults do not
count (reason NoClickHouseAccess).
- A shard is backed up once any of its replicas made the backup, AllReplicas
included; retention takes a backup as complete per shard.
- Restore follows the Altinity usual backup workflow: download
--hardlink-exists-files on every host, restore --schema=1 on every replica - the
first replica of the first shard with RESTORE_SCHEMA_ON_CLUSTER={cluster} -,
restore --data=1 on one replica per shard, then delete local. Every restore
command takes --ignore-dependencies; --rm, which recreates the databases of the
backup whole, only for an overwrite without tables; the filters (tables,
partitions, skipTables, skipTableEngines, databaseMapping, rbac) go on every
command. The
overwrite guard, the separate ON CLUSTER drop pass, the max_table_size_to_drop
checks and the dependency on the ZooKeeper configuration of the CHI are gone.
- The sidecar pins API_CREATE_INTEGRATION_TABLES and API_INTEGRATION_TABLES_HOST;
the operator reaches each host by its FQDN, port, scheme and cluster TLS.
- Events of the runs (Orchestrator*), conditions and reasons (Started, Failed,
FailedUnchanged, NoClickHouseAccess) are posted by the operator.
- Removed: backup.mode and pod mode, backup.orchestrator, the bash and Python
orchestrator scripts and their ConfigMaps, orchestratorConfigMapRef, jobName and
cronJobName, concurrencyPolicy, backup verification (spec.verify, the Verified
condition and its reasons, the verification metric), backup.path of the operator
configuration, and the RBAC rights for Jobs, CronJobs, Pods and RBAC objects.
- Names: {shard} of a naming template is the shard macro, {shardNum} the shard_num
from 1; restore keeps the naming prefix.
- The sidecar runs with TZ=UTC; restore_distributed_cluster defaults to {cluster}.
- spec.timeouts of backups, restores and schedules, with defaults in the operator
configuration (backup.timeouts) and in code: actionTimeout (12h),
unreachableTimeout (10m), httpReceiveTimeout (4h), errorResponsesMax (5), as Go
durations. Errors and empty polls are counted apart, with the response code.
- A failed backup deletes only its local copy, once; a failed restore keeps its
downloads, deleted only after success.
- Retention: keepBackups pins {name, until}; backups held by a schedule change, and
pins without until of such backups, expire by the schedule before the change
(backup time + keep x period of its longest entry that kept it), listed in
heldBackups with expires; an incomplete backup is deleted only when no
create_remote or upload of it runs on its shard per system.backup_actions.
- Storage paths: path and object_disks_path, macros expanded per host, must be
unique among the writers of one backup (AllReplicas: every replica), embedded
backups included; otherwise the backup or schedule is Aborted with
StoragePathConflict before any command.
- Finalizers: deleting an unfinished backup kills its commands and deletes its
local and remote copies; a completed one keeps its remote backup; a restore and a
schedule run are waited for.
- spec.taskID: a completed backup runs again under a new name, an unfinished one
under its own; a failed restore runs again, resuming its downloads
(spec.resumeDownload, default true) or starting over; an aborted schedule is
validated again. A taskID during a run is ignored with a warning.
- Restore: spec.replicatedCopyToDetached (default true) - the other replicas of a
shard restore the data of replicated tables to detached first, then one replica
attaches it; false downloads only the schema on them.
- A restore of access entities holds the backups and schedules of its own clusters
only; restores of different clusters run at the same time.
- Manifests regenerated with the current yq: plain <<: merge keys.
- CRDs, manifests, Helm chart, docs/backup.md and the examples follow; unit tests
cover the runs on fake sidecars; e2e test_010087, test_010088 and test_010089
(retention across a schedule change, keepBackups) cover real sidecars.
- ci: actions/setup-python@v7, which runs on Node.js 24.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Slach
force-pushed
the
backup-config-file
branch
from
November 18, 2026 10:07
eaf9557 to
a9e00ff
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix #2146
What changes
The operator itself runs backups and restores. There is no orchestrator script, no Job, no CronJob and no backup Pod: the operator drives the
clickhouse-backupsidecar of every host through SQL on the integration tables of that host's ClickHouse, as the Altinity usual backup workflow does.Transport: sidecar only. Every Pod of a backup-enabled cluster gets a
clickhouse-backup serversidecar. Its configuration, generated by the operator, always turns onapi.create_integration_tables, sosystem.backup_actionsandsystem.backup_listreadhttp://127.0.0.1:7171. The sidecar runs withTZ=UTC, andrestore_distributed_clusterdefaults to{cluster}.backup.modeand pod mode are removed.Credentials. The operator connects to each host by its FQDN, port, scheme and the cluster TLS, with the ClickHouse credentials of its own configuration (
clickhouse.accessor its Secret). They must be set explicitly: the built-in defaults do not count, and without them a backup or restore waits with reasonNoClickHouseAccess.Commands. A command is a row inserted into
system.backup_actionsof the host afternow('UTC')of that host (async_insert = 0,http_receive_timeout/http_send_timeoutfromhttpReceiveTimeout), and polled every 5 seconds, matched by its exact command text and start time, until it finishes. A busy sidecar (another operation is currently running) is retried with exponential backoff from 5 seconds up to 5 minutes. After an operator restart the leader resumes every run from its status: a running command is followed, a finished one is settled.Timeouts.
spec.timeoutsof aClickHouseBackup,ClickHouseRestoreandClickHouseBackupSchedule, defaults in the operator configurationbackup.timeoutsand in code, as Go durations:actionTimeout(12h),unreachableTimeout(10m),httpReceiveTimeout(4h),errorResponsesMax(5). Errors and empty polls are counted apart; the logs carry the response code.Backup.
create_remote --rbacon one replica per shard (replicaSelection: FirstPerShard,preferredReplica, fallback to another replica reported byOrchestratorReplicaFallback) or on every replica (AllReplicas). A shard is backed up once any of its replicas made the backup. A failed backup deletes only its local copy, once, and reports a remote leftover (OrchestratorFailedBackupLeft,OrchestratorFailedBackupRemoteLeft) with the SQL to delete it by hand.Storage paths. Before any command,
pathandobject_disks_pathof every writer of a backup, with the macros of each host expanded, must be unique (AllReplicas: every replica; embedded backups and CAS disks included). Otherwise the backup or schedule isAbortedwithStoragePathConflict, e.g.storage path 's3://backups/backup/main/0' is not unique: replicas ... both write a backup of one name there. Unknown macros give anOrchestratorStoragePathAdvicewarning.Backup names.
spec.backup.naming/clusters[].backup.naming, a template such asshard{shardNum}-{YYYYMMDDHHIISS}with{shard}(the shard macro),{shardNum}(shard_num, from 1),{cluster},{chi},{namespace},{prefix}and the time tokens. Retention, the incremental base and restore parse names back with it, so backups named by the Altinity usual backup workflow on a shared remote path are recognized. Empty keeps the current names.Schedules.
ClickHouseBackupSchedule.spec.schedules[](minute,hour,6hours,daily,weekly,monthly, withtime,day,keep,incremental,configOverrides) andspec.timeZone:ClickHouseBackupis created per run: a run lives instatus.activewhile it runs and instatus.recentRunsafterwards, bounded bysuccessfulRunsHistoryLimit(3) andfailedRunsHistoryLimit(1);startingDeadlineSecondsskips a slot missed by more; runs of one cluster never overlap;ClickHouseBackupwithscheduleRefruns the schedule now: its incremental entry, retention after it, a base of the next incremental backup.Retention follows the Altinity usual backup workflow: each entry keeps its own
keepnewest backups (GFS); the newest backup is always kept; run-now backups live within the window of the scheduled ones; an incomplete backup is deleted only once a newer complete one exists andsystem.backup_actionsshows nocreate_remoteoruploadof it on its shard; nothing is deleted during a restore; incremental bases are rebased byclickhouse-backup(forcedbackups_to_keep_remote: 0,rebase_*: true).Schedule changes.
spec.retention.onScheduleChange: Hold(default) records a change of the entries or the time zone instatus.retention.previous[]with itsexpires. The backups the schedule kept before it and the current one would delete are held, listed instatus.clusters[].heldBackupswithexpires: backup time +keep× the period of the longest old entry that kept it. Expired holds are deleted (OrchestratorRetentionHeldExpired), an expired change leavesprevious[](RetentionHeldExpired).keepBackups: [{name, until}]pins backups: withuntiluntil then; without it a backup held by a change until its expiry, any other backup while it is listed. Pins protect incomplete backups too.releaseHeld: <changeID>releases a change and all earlier ones,Pruneapplies a change at once.Config overrides.
schedules[].configOverridesandClickHouseBackup.spec.configOverrides: up to 16KEY=VALUEclickhouse-backup settings for one backup, sent ascreate_remote --env=.... Keys the operator pins (TZincluded),API_*, paths, buckets, endpoints, the embedded backup disk and secrets are refused.Restore, as in the Altinity usual backup workflow:
download --hardlink-exists-fileson every host (an incomplete local copy is resumed);restore --schema=1 --ignore-dependencieson every replica, the first replica of the first shard withRESTORE_SCHEMA_ON_CLUSTER={cluster}; skipped forpartitions;spec.replicatedCopyToDetached(default true): the other replicas of each shard runrestore --data=1 --ignore-dependencies --replicated-copy-to-detachedat the same time, so the parts of replicated tables are already local; with false they download only the schema;restore --data=1 --ignore-dependencieson one replica per shard, the next answering replica when the first does not;delete localon every host after success; a failed restore keeps its downloads.Filters go on every command:
tables,partitions,skipTables,skipTableEngines,databaseMapping,rbac.overwrite: truewithouttablesadds--rm, which recreates the databases of the backup as a whole; withtables, or withoutoverwrite, only the tables of the backup are recreated (clickhouse-backupdrops the existing ones) and no database is dropped. A restore of access entities holds the backups and schedules of its own clusters only; restores of different clusters run at the same time.Reruns:
spec.taskIDon all three kinds, withstatus.taskID,taskIDsStarted,taskIDsCompleted. A completed backup runs again under a new name (the old one instatus.previousBackupNames), an unfinished one under its own name after its earlier copies are deleted. A failed restore runs again:spec.resumeDownload(default true) resumes its earlier downloads, false deletes them and starts over. An aborted schedule is validated again. AtaskIDset while a run is active is ignored with aTaskIDIgnoredwarning.Deletion. Finalizers on
ClickHouseBackup,ClickHouseRestoreandClickHouseBackupSchedule. Deleting an unfinished backup kills only its own running commands (kill "<command>") and deletes its local and remote copies; a completed one keeps its remote backup for retention. A restore and a schedule run are waited for, and a deleted schedule starts no new slot. Unreachable hosts are given up afterunreachableTimeout, with aDeletionLeftoverwarning and the SQL to finish by hand.Validation. Backup enabled without
configSecretRef, a missing or invalid storage Secret, or an invalidbackupblock stops the CHI reconcile: statusAborted, reasonBackupConfigInvalid, Warning eventBackupStorageConfigInvalid.Status and events.
status.clusters[].runof aClickHouseBackup/ClickHouseRestorerecords the step, the hosts and every command sent with its status. PhasesPending,Running,Completed,Failed,Aborted. The operator posts every event itself (Orchestrator*per step and command) on theClickHouseBackup,ClickHouseBackupScheduleandClickHouseRestore.Removed:
backup.modeand pod mode,backup.orchestrator, the bash and Python orchestrator scripts and their ConfigMaps,orchestratorConfigMapRef,backup.pathof the operator configuration,jobName/cronJobName,concurrencyPolicy, backup verification (spec.verify, theVerifiedcondition and its metric), the restore overwrite guard, and the operator RBAC rights for Jobs, CronJobs, Pod writes, ServiceAccounts, Roles and RoleBindings. The operator gainsupdate/patchon its three backup kinds for the finalizers.Manifests are regenerated with the current yq, which writes plain
<<:merge keys in place of!!merge <<:; the documents are the same.Docs:
docs/backup.md(how the operator runs an operation, naming, timeouts, storage paths, migration from the Altinity usual backup workflow, schedule changes, reruns, deletion, config overrides, integration tables),docs/custom_resource_explained.md,docs/operator_configuration.md,docs/chb-examples/.Tests
go build ./... && go test ./...,-raceforpkg/controller/chbackupandpkg/model/chbackup: runs against fake sidecars (command text, polling, timeouts, resume after restart, busy backoff, replica fallback, copy to detached, finalizers, taskID reruns, path uniqueness, per-cluster gates), retention plans, schedule changes and their expiry, naming, slots, incremental base.helm templateof the regenerated chart.test_010087: backup, schedule, retention and restore of two clusters through real sidecars;test_010088: recovery of a new CHI from another one's backup;test_010089: retention across a schedule change: held backups expiring by the old schedule,keepBackupspins withoutuntil, withuntilin the past and in the future.Related, in
clickhouse-backup:/metricson its own address, so the API can stay on127.0.0.1;start/finishofsystem.backup_actionswithout a time zone (worked around here withTZ=UTC);system.backup_actions;DROP ... ON CLUSTERunderRESTORE_SCHEMA_ON_CLUSTERandmax_table_size_to_drop.deploy/operatorhubis not regenerated (done at release).dev/run_code_generator.shdoes not build with the current Go (#2152); deepcopy was generated with deepcopy-gen v0.34.1.🤖 Generated with Claude Code