Skip to content

postgres: build contrib extensions (pgcrypto, pg_trgm, ...) and enable lz4 - #1203

Merged
msample merged 2 commits into
mainfrom
postgres-contrib-lz4
Oct 30, 2026
Merged

msample merged 2 commits into
mainfrom
postgres-contrib-lz4

Conversation

@msample

@msample msample commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Summary

Make PostgreSQL's bundled extensions, the contrib modules, available by building them into the postgres package instead of adding separate per-extension packages. Starts from the request for pgcrypto and pg_trgm. Also enables --with-lz4, which resolves the TODO in build.sh now that lz4 is packaged.

Why bundle rather than split into packages like postgres-pgcrypto:

  • Same tarball, same configured build tree. A separate package would download, configure and build the same source again, and could drift from the server version. Bundled, it can't.
  • Cheap. In a trial build all of contrib added ~4.7 MB (269 files) to a ~35 MB package.
  • No new runtime deps. pgcrypto.so NEEDs only libcrypto.so.4 and libz.so.1, and postgres already depends on both.
  • Changes no behaviour. An extension does nothing until someone runs CREATE EXTENSION.
  • Matches other distros. nixpkgs (world/install-world), Debian since PG 10, Arch, Homebrew and the official Docker image all ship contrib with the server.

Extensions from outside the PostgreSQL tree (pgvector, PostGIS, ...) should still be separate packages that depend on postgres at runtime. That's out of scope here.

Changes

  • build.sh: make world-bin / make install-world-bin. That's the server plus contrib without the docs, PostgreSQL's documented target for this.
  • build.sh: fail the build if pgcrypto.so or sslinfo.so is missing. Contrib skips both silently when configure doesn't find OpenSSL.
  • build.sh / build.ncl: --with-lz4, with lz4 in runtime_deps alongside zstd. LZ4 becomes available for TOAST, WAL and backup compression. pglz stays the default, so existing behaviour doesn't change.
  • build.ncl: new standalone tests. Each starts its own throwaway server through a small server_test helper and does one job:
    • contrib_extensions_create: every available extension can be created (CASCADE), which loads each library.
    • pg_trgm: similarity(), and the planner uses a trigram GIN index for LIKE '%abc%'.
    • pgcrypto: digest (known SHA-256 vector), gen_random_bytes, crypt/gen_salt('bf'), pgp_sym_*, raw AES encrypt/decrypt.
    • pgcrypto_legacy_provider: raw Blowfish works when OPENSSL_CONF activates the OpenSSL legacy provider.
    • lz4_compression: a COMPRESSION lz4 column is stored as lz4, and wal_compression = lz4 is accepted.

Outputs are unchanged: the existing globs already cover everything contrib installs. The four contrib/spi *.example docs under usr/share/doc aren't captured, same as before, since nothing captures docs.

Checklist

  • I've read CONTRIBUTING.md.
  • I've accepted the ICLA (and CCLA if contributing on my employer's time). CLA Assistant will prompt me on this PR if I haven't already.
  • min check passes for the affected packages/harnesses.
  • min patched-build <name> succeeds for any package I added or modified.
  • For new packages: n/a
  • For version bumps: n/a (stays on 18.6)

Notes for reviewers

  • Tested on aarch64 only. Patched build, then min check with all six standalone tests passing, the existing initdb_c_utf8 included. To confirm the new tests aren't vacuous, I flipped one assertion and checked that min check failed and named lz4_compression. x86_64 is covered by CI only.
  • Reproducibility (checked on aarch64). I built this PR's exact build.sh twice, with an uncommitted probe appended that hashes $OUTPUT_DIR. A marker that differed between the two runs forced two real builds instead of a cache hit; both did the full compile (1,566 gcc invocations each) and configure found LZ4. The two output trees were identical: all 2,028 entries (1,924 files, 9 symlinks, 95 directories) had the same type, mode, symlink target and sha256. I couldn't run the host-side mip --no-cache --rebuild from the sandbox, and I haven't run this check on x86_64.
  • Legacy ciphers. Raw encrypt()/decrypt() with bf, des or cast5 fail by default, because the shipped openssl.cnf doesn't activate the legacy provider. That's the same on every OpenSSL 3+/4 distro, and upstream documents it in pgcrypto's notes. crypt() with bf/md5/xdes uses pgcrypto's built-in code and works either way. The new test covers the opt-in path rather than enabling legacy ciphers globally.
  • Not built with the current configure flags: uuid-ossp (needs --with-uuid; mostly replaced by the built-in gen_random_uuid()/uuidv4()/uuidv7()), xml2, sepgsql, and the plperl/plpython transforms, which would make perl/python runtime deps of postgres. Left for a follow-up if anyone asks.
  • libs output. The output glob is usr/lib/**, so libpq and the server modules live in one output, and contrib adds 49 .so files to it. Nothing depends on postgres today. If a libpq-only consumer appears, that output is worth splitting.
  • About the tests. min check reports which test failed but not its output, which is why the helper keeps one concern per test.

🤖 Generated with Claude Code

…e lz4

Build world-bin so the server ships the contrib modules from its own
release tarball, compiled in the same configured tree, so they can never
drift from the server version. With the current configure flags that is
52 modules (pgcrypto, pg_trgm, hstore, citext, pg_stat_statements,
postgres_fdw, ...). In a trial build they added ~4.7 MB to a ~35 MB
package and no runtime deps: pgcrypto links libcrypto and libz, which
postgres already depends on. An extension does nothing until someone runs
CREATE EXTENSION.

Configure --with-lz4 now that lz4 is packaged, resolving the TODO in
build.sh: LZ4 for TOAST, WAL and backup compression. pglz stays the
default, so existing behaviour does not change.

contrib skips pgcrypto and sslinfo silently when configure finds no
OpenSSL, so build.sh now fails if either is missing.

New standalone tests each start a throwaway server: every available
extension can be created; pg_trgm similarity and a trigram GIN index
serving LIKE; pgcrypto digest, random bytes, bcrypt, PGP and AES; raw
Blowfish through the OpenSSL legacy provider opt-in; and lz4 column and
WAL compression.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@msample
msample requested a review from a team as a code owner October 30, 2026 05:00
@coderabbitai

coderabbitai Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

  • Run on-demand review

This review includes 2 billable files and costs up to $0.50.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 47 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 61 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

Run configuration
Commits

Reviewing files that changed from the base of the PR and between e75fe4b and 06e5ab6.

Files selected for processing: 2
  • packages/postgres/build.ncl
  • packages/postgres/build.sh

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 2e963a5 and e75fe4b.

Files selected for processing: 2
  • packages/postgres/build.ncl
  • packages/postgres/build.sh

Included review availability: This review used your included allowance. 4 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

Walkthrough

The PostgreSQL build enables LZ4, builds and installs world-bin, and checks for installed pgcrypto and sslinfo modules. New server tests cover extension creation, trigram indexing, cryptographic operations, and LZ4 compression.

Changes: PostgreSQL build and server tests

Layer / File(s) Summary
Build configuration and installation
packages/postgres/build.sh, packages/postgres/build.ncl
The build enables LZ4, uses world-bin build and install targets, adds LZ4 as a runtime dependency, and checks for installed pgcrypto.so and sslinfo.so.
Server test harness and coverage
packages/postgres/build.ncl
A shared helper initializes, starts, and cleans up a temporary PostgreSQL server. Tests cover available extensions, pg_trgm similarity and GIN index use, pgcrypto operations including legacy-provider encryption, and LZ4 column and WAL compression.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Possibly related issues

  • gominimal/pkgs#1149: This issue calls for behavioral standalone package tests, and this change adds standalone PostgreSQL server tests.

Possibly related PRs

  • gominimal/pkgs#1140: The legacy-provider test uses OpenSSL configuration and provider modules, which are affected by this PR’s OpenSSL package changes.

Suggested reviewers: bryan-minimal

Merge Risk: Minimal · previous review e75fe

No concrete merge-blocking risk was identified in the build or server tests.

Pre-merge checks: Passed 5
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely summarizes the main changes: building PostgreSQL contrib extensions and enabling LZ4 support.
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files. (1 skipped: 1 unsupported.)

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@norrietaylor norrietaylor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks correct and bundling contrib into postgres is the right call. Two things need doing before merge.

Blocking

  1. x86_64 hasn't been built or tested. The PR notes say x86_64 is covered by CI, but PR CI only runs static checks. In the minimal-check log for postgres, standalone tests, output types valid, missing runtime_deps and enumerate bins all show Skip. So the five new tests, the world-bin build and the --with-lz4 link have run only on one aarch64 machine. Please do an x86_64 patched build and run min check on it.
  2. Reproducibility needs re-checking on the final change. The double build was done with make -C contrib, before the switch to world-bin and --with-lz4. Please do two --rebuild builds of the head commit and diff the two $OUTPUT_DIR trees.

Suggestions (non-blocking)

  • build.sh: a bare test -e fails the build without saying why. Something like || { echo "pgcrypto.so missing: configure did not find OpenSSL" >&2; exit 1; } would make the failure clear.
  • runtime_deps: lz4 brings in the whole package (CLI tools, man pages, headers) when postgres only needs liblz4. It's consistent with how zstd and zlib are done today, so fine for this PR. Moving to subsetOf later would also mean adding lz4 to build_deps so configure still finds the headers.
  • pgcrypto_legacy_provider: this mostly tests the openssl package, i.e. that legacy.so is where OpenSSL's MODULESDIR expects it. A comment saying so would help whoever sees it fail after an openssl layout change.
  • contrib_extensions_create: a short comment on why the threshold is > 40 (contrib currently gives about 49) would help.

Checked and fine

  • No accidental %{ interpolation in the Nickel multiline strings.
  • The bcrypt $h substitution into q is safe.
  • A failure inside for e in $(q …) is still caught by the count check that follows.
  • The TOAST value is large enough that the pg_column_compression = lz4 check really tests compression.
  • wal_compression = lz4 is rejected if the build lacks lz4, so that check is meaningful.
  • The existing output globs cover everything contrib installs.

@norrietaylor
norrietaylor self-requested a review October 30, 2026 05:35
@msample

msample commented Oct 30, 2026

Copy link
Copy Markdown
Member Author

Reproducibility check on arm64 (aarch64): reproducible.

I built this PR's build.sh (commit e75fe4b) twice and compared the two $OUTPUT_DIR trees:

  • Two real builds. A one-word marker that differed between the runs forced both to compile from scratch instead of hitting the cache. Each ran the full compile (1,566 gcc invocations), and configure found LZ4 both times.
  • Identical output. All 2,028 entries (1,924 files, 9 symlinks, 95 directories) had the same type, mode, symlink target and sha256. That includes usr/bin/postgres, libpq.so/libpq.a, and the contrib modules such as pgcrypto.so and pg_trgm.so.

The hashing step was appended only on a scratch branch and isn't part of this PR. This was done in-sandbox with min package patched-build, not the host-side mip --no-cache --rebuild. x86_64 hasn't been checked for reproducibility.

@msample
msample enabled auto-merge October 30, 2026 06:16
@msample
msample added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit af8e459 Oct 30, 2026
13 checks passed
@msample
msample deleted the postgres-contrib-lz4 branch October 30, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants