Skip to content

create: run the first FREEZE serially when shadow/increment.txt doesn't exist - #1674

Open
Slach wants to merge 3 commits into
masterfrom
fix/serial-first-freeze-on-fresh-server
Open

Slach wants to merge 3 commits into
masterfrom
fix/serial-first-freeze-on-fresh-server

Conversation

@Slach

@Slach Slach commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Reopened from #1664 by @RyoKusnadi (original fork was deleted); commit cherry-picked onto current master with authorship preserved.

Problem

On a fresh clickhouse-server, the first create / create_remote over several tables can fail with code: 32, File .../shadow/increment.txt is empty. You must fill it manually. It happens once per server lifetime, but it is a failed backup (and a flaky CI run).

How it happens: a new server has no shadow/increment.txt. create freezes tables in parallel (clickhouse.max_connections workers). Two FREEZE queries race to create the file: one creates it empty, the other locks it first and finds it empty (the CounterInFile race described in #1651).

Cause

createBackupLocal in pkg/backup/create.go starts all table workers at once, so the first FREEZEs overlap.

Fix

Before starting the workers, serialFreezeTableIndex checks <default disk path>/shadow/increment.txt. If it is missing (or can't be checked, for example clickhouse-backup runs on another host), the first table that will really run FREEZE is backed up serially, then the rest run in parallel as before. I don't write the file myself, since it belongs to clickhouse-server. The per-table body moved into a local closure so the serial and parallel paths share it.

Scope

  • Not changed: retry on other transient FREEZE errors, a FREEZE-only concurrency limit, pre-warming increment.txt in testflows (open questions in the issue).
  • use_embedded_backup_restore doesn't use FREEZE and is unaffected.
  • clean --all already keeps increment.txt.

Testing

TestSerialFreezeTableIndex (temp dirs, no ClickHouse): fresh dir, unreachable path, existing file, doBackupData=false, skipped/schema-only tables, empty list.

  • Before: undefined: serialFreezeTableIndex (test written first).
  • After: go test ./pkg/backup/ -run TestSerialFreezeTableIndex passes; go test ./pkg/backup/ passes (376 tests); go vet ./pkg/backup/ and gofmt -l are clean.
  • Not run: integration/testflows tests (need a live ClickHouse), so the race itself is not reproduced here.

Risk

Only the first backup on a server without increment.txt is a bit slower (one serial FREEZE). When the file can't be checked remotely, every create has one serial FREEZE first.

Fixes #1651

🤖 Generated with Claude Code

RyoKusnadi and others added 3 commits November 9, 2026 11:44
…'t exist, fix #1651

Signed-off-by: Ryo Kusnadi <ryokusnadi@gmail.com>
Follow-up for #1651 review findings:
- serialFreezeTableIndex is a Backuper method sharing isFreezeEngine and
  isTableOnPlainDisks with AddTableToLocalBackup, so a View, Distributed,
  Kafka or plain-disk table is never chosen for the serial step; the
  smallest eligible table is chosen (GetTables sorts by size DESC), an
  empty table only without freeze_by_part (ALTER TABLE FREEZE increments
  the counter before iterating parts, FREEZE PARTITION runs per part)
- an empty shadow/increment.txt fails every FREEZE in ClickHouse
  CounterInFile (only a missing file is initialized), remove it and run
  the first FREEZE serially so clickhouse-server recreates it
- create_remote --streaming gets the same serial-first step
- name the table in the serial error and log, unit tests cover engines,
  plain disks, freeze_by_part and the empty counter, add integration
  TestSerialFirstFreezeOnFreshServer and a ChangeLog entry

The counter is global (<path>/shadow/increment.txt in
MergeTreeData::freezePartitionsByMatcher), not per disk, so only the
default data path is checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DROP DATABASE ... SYNC is a syntax error on ClickHouse 1.1.54394, 19.17,
20.3 and 20.8, so TestSerialFirstFreezeOnFreshServer failed in its
cleanup step on those CI versions after all backup checks had passed.
env.dropDatabase adds SYNC only for Atomic/Replicated databases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

create: parallel FREEZE on fresh server fails with 'shadow/increment.txt is empty' (CounterInFile first-create race)

2 participants