Skip to content

icu: generate all of ICU's data from text (no prebuilt .dat or property tables) - #836

Merged
bryan-minimal merged 5 commits into
mainfrom
bryan/icu-data-from-text
Nov 16, 2026
Merged

bryan-minimal merged 5 commits into
mainfrom
bryan/icu-data-from-text

Conversation

@bryan-minimal

@bryan-minimal bryan-minimal commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

What

packages/icu (78.3) built from icu4c-78.3-sources.tgz, which ships ICU's data prebuilt:

  • data/in/icudt78l.dat (33 MB): the whole locale, collation, break-iterator and converter database;
  • 21 core Unicode property files: data/in/*.icu, *.nrm, coll/*.icu, plus the *_props_data.h,
    propname_data.h and norm2_nfc_data.h headers compiled into libicuuc.

make only repackaged these into libicudata/libicuuc. They reach every ICU consumer at runtime,
including node/node-lts (--with-intl=system-icu). A content census of node's whole closure found them
to be the only prebuilt binaries that reach its installed output.

This PR builds ICU from the release-78.3 git tag, where all of this exists as text, and generates
every data file in the build:

  1. It deletes the checked-in generated files, as ICU's own icu4c/source/data/unidata/generate.sh does,
    and fails if a .dat is present.
  2. It builds ICU's generators (gennorm2, genprops, genuca) from ICU's BUILD.bazel files,
    without Bazel. minibazel.py evaluates those files, which use only plain
    cc_library/cc_binary, and compiles the targets with the toolchain gcc/g++.
  3. It runs them as generate.sh does: normalization data from unidata/norm2/*.txt, properties from
    ppucd.txt, and collation from FractionalUCA.txt.
  4. The normal ICU build then compiles the locale, collation and converter data from text with
    genrb/makeconv/genbrk/… (4,073 resource bundles, 190 converters).

Evidence (lab, cg-fresh amd64)

  • ICU's generate.sh with Bazel at the tag: all 22 regenerated files are byte-identical to the
    shipped ones.
  • The same steps without Bazel (minibazel.py): all 22 are byte-identical.
  • Data from text: with no .dat present, the built icudt78l.dat is byte-identical to the shipped
    one (all 4,305 items).
  • This recipe vs the current one: libicudata, libicuuc and libicui18n are byte-identical
    (253/253 installed files). The one exception is usr/lib/icu/78.3/pkgdata.inc, which records the
    build's CFLAGS, including the source path.

Notes

  • Text inputs still trusted, all upstream text in the tag:
    • ppucd.txt, ICU's preparsed form of Unicode's UCD, made by tools/unicode/py/preparseucd.py;
      regenerating it from the raw UCD is a possible follow-up;
    • FractionalUCA.txt from CLDR;
    • the normalization and locale sources.
  • New build dep: python (no cycle; python does not depend on icu).
  • Source: the tag tarball, mirrored to gs://minimal-staging-archives/icu-release-78.3.tar.gz
    (sha256 f06bcab7…, read back).
  • Importers cold-rebuild: actions-runner, libxml2, node, node-lts, postgres, re2. Their ICU bytes are
    unchanged.

Refreshed 2026-11-13

  • Merged origin/main (626 commits) into the branch: 85b9fdf4, no conflicts. The PR's diff is still
    only packages/icu/{build.ncl,build.sh,minibazel.py}, and main's icu is still 78.3.
  • Mirror re-verified. The anonymous stream of icu-release-78.3.tar.gz hashes to f06bcab7…3fd931, the
    pinned sha256. GitHub's tag archive for release-78.3 (gh release download --archive tar.gz) hashes to
    the same value, so the mirror is byte-identical to the tag. dd053f27 records that archive URL as a comment
    above the source.
  • Updater (pkgmgr-rs): the next release bumps without a marker. This recipe has the same shape as age
    and act: a gs:// mirror of a GitHub tag archive templated on %{version}.
    • The GitHub tier finds release-79.1; supply-chain's normalize strips release-. Its
      tarball_url is archive/refs/tags/release-<v>.tar.gz.
    • The release-asset swap looks for an asset named icu-release-<v>.tar.gz. No release has one, so the
      tag archive is kept.
    • derive_update_plan mirrors it to gs://minimal-staging-archives/icu-release-<v>.tar.gz. Both sides
      are .tar.gz and the size is about the same as 78.3's, so the extension and 2× size guards pass.
    • The comment URL is rewritten along with the version.
    • Tag format: tags are dotted since 78 (release-78.1 … release-79.1) and 77 was the last dashed one
      (release-77-1). The archive's top directory is icu-release-<tag minus "release-">, which matches
      build.sh's icu-release-${MINIMAL_ARG_VERSION}.
    • Caveat: the layout guard can't see that directory, because build.sh runs cd "$C", not
      cd …${MINIMAL_ARG_VERSION}. A future dashed tag would fail at extraction, not in the updater.
  • ICU 79.1 is held: Update icu to 79.1 #1139 was closed until node/node-lts support ICU 79. When it reopens, the bump
    should go through this path; minibazel.py against 79's BUILD.bazel files is untested.
  • Overlap with cpio, unzip, icu, libxml2, libxslt from staging-archives; gcc-15.2.0-glibc declares libzstd #1145: that draft repoints packages/icu/build.ncl's source at
    gs://…/unicode-org/icu/icu4c-%{version}-sources.tgz. This PR supersedes cpio, unzip, icu, libxml2, libxslt from staging-archives; gcc-15.2.0-glibc declares libzstd #1145's icu hunk: drop that hunk
    from cpio, unzip, icu, libxml2, libxslt from staging-archives; gcc-15.2.0-glibc declares libzstd #1145, or rebase it out, whichever lands second.

🤖 Generated with Claude Code

bryan-minimal and others added 3 commits October 7, 2026 14:39
The sources tarball ships the data prebuilt (icudt78l.dat). Building from
icu4c-78.3-data.zip with the ICU tools built here gives a byte-identical
archive and libicudata (all 4305 items).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ICU's release tarballs also ship its core property tables prebuilt
(data/in/*.icu, *.nrm, coll/*.icu and the *_props_data.h headers compiled into
libicuuc). The release-78.3 tag carries their text inputs and generators;
minibazel.py builds gennorm2, genprops and genuca from ICU's BUILD.bazel files
with gcc, and the recipe runs them as ICU's data/unidata/generate.sh does.
Every installed library is byte-identical to the previous recipe's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

This review includes 3 billable files and costs up to $0.75.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 59 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 84 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between ded4f65 and dd053f2.

Files selected for processing: 3

  • packages/icu/build.ncl
  • packages/icu/build.sh
  • packages/icu/minibazel.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

bryan-minimal and others added 2 commits November 13, 2026 13:49
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The updater bumps this comment's version with the source (rewrite_comment_url_versions),
so a reviewer can check the next mirror against GitHub.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@norrietaylor norrietaylor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scope matches the title. The PR diff is only packages/icu/{build.ncl,build.sh,minibazel.py}. Clobber none: git log -- packages/icu shows only this branch's own commits on top of the 78.3 bump (#62) and the smoke-test batch (#1368). The smoketest from #1368 is still in build.ncl. Correctness The python build dep adds no cycle. packages/python/build.ncl does not import icu or any of its importers. The .dat guard runs after the rm -f, so a prebuilt archive in the tree stops the build. No other package in the repo has a Bazel evaluator to reuse. Two points are for the next bump, not for this build. First, minibazel.py silently drops rule attributes it does not model. Second, the package test does not touch any of the data that this PR now generates. The importer list in the PR body omits node-24, which also imports ../icu/build.ncl. Not checked: the BUILD.bazel files at release-78.3, because the sandbox denied network access. I also did not check the byte-identical claims, because they need a build. I relied on the lab evidence in the PR body for both.

Comment thread packages/icu/minibazel.py
def rule(kind):
def f(name, **kw):
rules["//%s:%s" % (pkg, name)] = dict(kw, kind=kind, pkg=pkg, name=name)
return f

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low · rule() accepts any keyword argument, but the build reads only srcs, deps, includes, local_defines and linkopts. A BUILD.bazel that adds defines, copts or implementation_deps is evaluated without error, and that attribute is dropped. The generators then compile with different flags or miss a dependency. They can emit data that differs from upstream, and that data ships in libicuuc/libicudata. The PR body says the 79 bump will run minibazel.py against BUILD files it was not tested on. Make rule() fail on any keyword outside the modeled set, so a new attribute stops the build in place of a silent change. The dedupe at line 87 has the same issue: a source listed by two rules gets only the first rule's local_defines. A check that the defines match can make that case fail too.

Comment thread packages/icu/build.sh
"$T/genuca" --hanOrder implicit "$S/icu4c"
"$T/genuca" --hanOrder radical-stroke "$S/icu4c"
"$T/genuca" --icu4x --hanOrder implicit "$S/icu4c"
"$T/genuca" --icu4x --hanOrder radical-stroke "$S/icu4c"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low · This PR moves every ICU data file to in-build generation. The package test is still only uconv --version (build.ncl:60), and that test reads none of this data. Today the byte-identity evidence exists only in the lab. On the next bump, a generator or minibazel.py regression can produce wrong normalization, property or collation data, and no package test will fail. Add a standalone test that loads the generated data. Examples: uconv -x any-nfkc on a known input, a converter round trip with uconv -f utf-8 -t iso-8859-1, and icuinfo to confirm that the data loads.

Comment thread packages/icu/minibazel.py
@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""icu-minibazel.py ROOT OUT TARGET...: build ICU's cc_binary targets from its own BUILD.bazel files, without Bazel.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit · The docstring names the script icu-minibazel.py, but the file is minibazel.py, and build.sh calls it as minibazel.py. Use the real name.

Suggested change
"""icu-minibazel.py ROOT OUT TARGET...: build ICU's cc_binary targets from its own BUILD.bazel files, without Bazel.
"""minibazel.py ROOT OUT TARGET...: build ICU's cc_binary targets from its own BUILD.bazel files, without Bazel.

@bryan-minimal
bryan-minimal added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 7773a9e Nov 16, 2026
15 checks passed
@bryan-minimal
bryan-minimal deleted the bryan/icu-data-from-text branch November 16, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants