Repository navigation
icu: generate all of ICU's data from text (no prebuilt .dat or property tables) - #836
Conversation
The sources tarball ships the data prebuilt (icudt78l.dat). Building from icu4c-78.3-data.zip with the ICU tools built here gives a byte-identical archive and libicudata (all 4305 items). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ICU's release tarballs also ship its core property tables prebuilt (data/in/*.icu, *.nrm, coll/*.icu and the *_props_data.h headers compiled into libicuuc). The release-78.3 tag carries their text inputs and generators; minibazel.py builds gennorm2, genprops and genuca from ICU's BUILD.bazel files with gcc, and the recipe runs them as ICU's data/unidata/generate.sh does. Every installed library is byte-identical to the previous recipe's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reached
This review includes 3 billable files and costs up to $0.75.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 59 minutes for your next included review. View limit details
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The updater bumps this comment's version with the source (rewrite_comment_url_versions), so a reviewer can check the next mirror against GitHub. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
norrietaylor
left a comment
There was a problem hiding this comment.
Scope matches the title. The PR diff is only packages/icu/{build.ncl,build.sh,minibazel.py}. Clobber none: git log -- packages/icu shows only this branch's own commits on top of the 78.3 bump (#62) and the smoke-test batch (#1368). The smoketest from #1368 is still in build.ncl. Correctness The python build dep adds no cycle. packages/python/build.ncl does not import icu or any of its importers. The .dat guard runs after the rm -f, so a prebuilt archive in the tree stops the build. No other package in the repo has a Bazel evaluator to reuse. Two points are for the next bump, not for this build. First, minibazel.py silently drops rule attributes it does not model. Second, the package test does not touch any of the data that this PR now generates. The importer list in the PR body omits node-24, which also imports ../icu/build.ncl. Not checked: the BUILD.bazel files at release-78.3, because the sandbox denied network access. I also did not check the byte-identical claims, because they need a build. I relied on the lab evidence in the PR body for both.
| def rule(kind): | ||
| def f(name, **kw): | ||
| rules["//%s:%s" % (pkg, name)] = dict(kw, kind=kind, pkg=pkg, name=name) | ||
| return f |
There was a problem hiding this comment.
low · rule() accepts any keyword argument, but the build reads only srcs, deps, includes, local_defines and linkopts. A BUILD.bazel that adds defines, copts or implementation_deps is evaluated without error, and that attribute is dropped. The generators then compile with different flags or miss a dependency. They can emit data that differs from upstream, and that data ships in libicuuc/libicudata. The PR body says the 79 bump will run minibazel.py against BUILD files it was not tested on. Make rule() fail on any keyword outside the modeled set, so a new attribute stops the build in place of a silent change. The dedupe at line 87 has the same issue: a source listed by two rules gets only the first rule's local_defines. A check that the defines match can make that case fail too.
| "$T/genuca" --hanOrder implicit "$S/icu4c" | ||
| "$T/genuca" --hanOrder radical-stroke "$S/icu4c" | ||
| "$T/genuca" --icu4x --hanOrder implicit "$S/icu4c" | ||
| "$T/genuca" --icu4x --hanOrder radical-stroke "$S/icu4c" |
There was a problem hiding this comment.
low · This PR moves every ICU data file to in-build generation. The package test is still only uconv --version (build.ncl:60), and that test reads none of this data. Today the byte-identity evidence exists only in the lab. On the next bump, a generator or minibazel.py regression can produce wrong normalization, property or collation data, and no package test will fail. Add a standalone test that loads the generated data. Examples: uconv -x any-nfkc on a known input, a converter round trip with uconv -f utf-8 -t iso-8859-1, and icuinfo to confirm that the data loads.
| @@ -0,0 +1,95 @@ | |||
| #!/usr/bin/env python3 | |||
| """icu-minibazel.py ROOT OUT TARGET...: build ICU's cc_binary targets from its own BUILD.bazel files, without Bazel. | |||
There was a problem hiding this comment.
nit · The docstring names the script icu-minibazel.py, but the file is minibazel.py, and build.sh calls it as minibazel.py. Use the real name.
| """icu-minibazel.py ROOT OUT TARGET...: build ICU's cc_binary targets from its own BUILD.bazel files, without Bazel. | |
| """minibazel.py ROOT OUT TARGET...: build ICU's cc_binary targets from its own BUILD.bazel files, without Bazel. |
What
packages/icu(78.3) built fromicu4c-78.3-sources.tgz, which ships ICU's data prebuilt:data/in/icudt78l.dat(33 MB): the whole locale, collation, break-iterator and converter database;data/in/*.icu,*.nrm,coll/*.icu, plus the*_props_data.h,propname_data.handnorm2_nfc_data.hheaders compiled intolibicuuc.makeonly repackaged these intolibicudata/libicuuc. They reach every ICU consumer at runtime,including
node/node-lts(--with-intl=system-icu). A content census of node's whole closure found themto be the only prebuilt binaries that reach its installed output.
This PR builds ICU from the release-78.3 git tag, where all of this exists as text, and generates
every data file in the build:
icu4c/source/data/unidata/generate.shdoes,and fails if a
.datis present.gennorm2,genprops,genuca) from ICU'sBUILD.bazelfiles,without Bazel.
minibazel.pyevaluates those files, which use only plaincc_library/cc_binary, and compiles the targets with the toolchain gcc/g++.generate.shdoes: normalization data fromunidata/norm2/*.txt, properties fromppucd.txt, and collation fromFractionalUCA.txt.genrb/makeconv/genbrk/… (4,073 resource bundles, 190 converters).Evidence (lab, cg-fresh amd64)
generate.shwith Bazel at the tag: all 22 regenerated files are byte-identical to theshipped ones.
minibazel.py): all 22 are byte-identical..datpresent, the builticudt78l.datis byte-identical to the shippedone (all 4,305 items).
libicudata,libicuucandlibicui18nare byte-identical(253/253 installed files). The one exception is
usr/lib/icu/78.3/pkgdata.inc, which records thebuild's CFLAGS, including the source path.
Notes
ppucd.txt, ICU's preparsed form of Unicode's UCD, made bytools/unicode/py/preparseucd.py;regenerating it from the raw UCD is a possible follow-up;
FractionalUCA.txtfrom CLDR;python(no cycle; python does not depend on icu).gs://minimal-staging-archives/icu-release-78.3.tar.gz(sha256
f06bcab7…, read back).unchanged.
Refreshed 2026-11-13
origin/main(626 commits) into the branch:85b9fdf4, no conflicts. The PR's diff is stillonly
packages/icu/{build.ncl,build.sh,minibazel.py}, and main's icu is still 78.3.icu-release-78.3.tar.gzhashes tof06bcab7…3fd931, thepinned sha256. GitHub's tag archive for
release-78.3(gh release download --archive tar.gz) hashes tothe same value, so the mirror is byte-identical to the tag.
dd053f27records that archive URL as a commentabove the source.
ageand
act: ags://mirror of a GitHub tag archive templated on%{version}.release-79.1; supply-chain'snormalizestripsrelease-. Itstarball_urlisarchive/refs/tags/release-<v>.tar.gz.icu-release-<v>.tar.gz. No release has one, so thetag archive is kept.
derive_update_planmirrors it togs://minimal-staging-archives/icu-release-<v>.tar.gz. Both sidesare
.tar.gzand the size is about the same as 78.3's, so the extension and 2× size guards pass.release-78.1…release-79.1) and 77 was the last dashed one(
release-77-1). The archive's top directory isicu-release-<tag minus "release-">, which matchesbuild.sh'sicu-release-${MINIMAL_ARG_VERSION}.build.shrunscd "$C", notcd …${MINIMAL_ARG_VERSION}. A future dashed tag would fail at extraction, not in the updater.should go through this path;
minibazel.pyagainst 79'sBUILD.bazelfiles is untested.packages/icu/build.ncl's source atgs://…/unicode-org/icu/icu4c-%{version}-sources.tgz. This PR supersedes cpio, unzip, icu, libxml2, libxslt from staging-archives; gcc-15.2.0-glibc declares libzstd #1145's icu hunk: drop that hunkfrom cpio, unzip, icu, libxml2, libxslt from staging-archives; gcc-15.2.0-glibc declares libzstd #1145, or rebase it out, whichever lands second.
🤖 Generated with Claude Code