Add OpenID Connect Core conformance tests to CI - #54854

It runs the OpenID Foundation conformance suite's OpenID Connect Core Config, Basic, Implicit and Hybrid certification plans against Keycloak with discovery and statically registered clients, as the OidcTestSuite matrix job of the conformance workflow. The tests live in their own package (oidc).

Machine

All conformance tests ran on one machine: Keycloak, the test JVM, and the conformance suite containers.

• CPU: Intel Core i7-1360P (13th Gen), 16 logical CPUs available to WSL2 and Docker
• Memory: 16 GB (15 GiB available to WSL2 and Docker)
• OS: Ubuntu 24.04.5 LTS on WSL2 (Windows 11), kernel 6.18.40.1-microsoft-standard-WSL2
• Software: Docker 29.9.3, OpenJDK 21.0.12, Maven 3.9.16, conformance suite release-v5.3.0

Executed tests and results

The tests run the OpenID Connect Core OP certification plans of the OpenID Foundation conformance suite against Keycloak. Each conformance module variant is one JUnit test with an expected result. Every module finished with its expected result.

OidcTestSuite and OidcFormPostTestSuite use discovery and statically registered clients with client_secret_basic and client_secret_post. The Form Post plans run the same modules as the Basic, Implicit and Hybrid plans, with the authorization response sent as a form post (response_mode=form_post), and they finish with the same results. In OidcDynamicTestSuite, the suite registers its own clients through Keycloak's anonymous dynamic client registration, with private_key_jwt client authentication.

The REVIEW modules finish correctly, but the suite asks a human to check a screenshot:

• oidcc-prompt-login and oidcc-max-age-1 (Basic, Implicit and Hybrid, plain and Form Post): the second login page must be shown again, and the suite takes a screenshot of it.
• oidcc-ensure-registered-redirect-uri (Basic, Implicit and Hybrid, plain and Form Post), and oidcc-ensure-redirect-uri-in-authorization-request, oidcc-redirect-uri-query-mismatch and oidcc-redirect-uri-query-added (Dynamic): Keycloak shows its error page ("Invalid parameter: redirect_uri") instead of redirecting to a redirect_uri that is not registered or is missing.
• oidcc-registration-logo-uri, oidcc-registration-policy-uri and oidcc-registration-tos-uri (Dynamic): the suite takes a screenshot of Keycloak's consent page, which shows the logo, privacy policy and terms of service the client registered.

All other modules finish with PASSED. No module finishes with WARNING. Without help, the suite would warn about Keycloak's ID tokens: the ID token always contains the typ claim, and it contains the email claims even when an access token is issued. The test realms therefore use a script based protocol mapper that removes the typ claim, and the email claims unless the response type is id_token. They also use Keycloak's claims parameter mapper.

In the Dynamic realm, Keycloak's default anonymous client registration policies are replaced, because their trusted hosts policy rejects the suite's registrations. The realm uses the consent required policy and a limit of 1000 clients instead. The registered clients get the same ID token mappers as the static clients, through a default client scope, and profile and email are optional scopes, as for the static clients.

In the Dynamic plan, Keycloak fetches URLs the suite publishes for its clients, such as jwks_uri, request_uri and sector_identifier_uri. The suite publishes them on its container-internal host name, which Keycloak, running on the Docker host, cannot resolve. The tests therefore send Keycloak's requests for that host through a tunnel in the test JVM, using the proxy-mappings option of Keycloak's outgoing HTTP client. For oidcc-server-rotate-keys, the test adds a new signing key to the realm after the module has fetched the realm's keys and before it starts the module.

Intentionally skipped tests

• oidcc-idtoken-unsigned (Dynamic, code and code token): the suite skips it, because Keycloak does not issue unsigned ID tokens (alg=none) and does not list none in id_token_signing_alg_values_supported. Servers are not required to support none.
• oidcc-discovery-endpoint-verification and oidcc-server-rotate-keys (Dynamic): they run in the code plan only, not in all six response type plans, because their behaviour does not depend on the response type.

No module of the covered plans is disabled.

Not covered:

• the 3rd Party-Initiated Login OP plan;
• the Basic, Implicit, Hybrid and Form Post plans with dynamic client registration or static server metadata variants (the Dynamic OP plan covers dynamic client registration);
• the OpenID Connect logout plans (RP-Initiated, Front-Channel, Back-Channel and Session Management), which are covered separately.

Time to complete

Each suite is one job of the conformance workflow matrix, so in CI the three suites run in parallel:

Time of each test class when run on its own, which includes compiling the module and starting Keycloak and the conformance suite (about 40–60 s):

Much of the time goes to the suite's headless browser logging in to Keycloak, so the tests turn off CSS processing in that browser (browsercontrol_css_enable=false).

On this machine, modules occasionally failed for environment reasons, which the automatic retries (see below) usually covered:

• the TLS connection from the suite to Keycloak through the Testcontainers host tunnel was dropped ("Remote host terminated the handshake");
• the suite's browser got stuck on about:blank.

In the per-class runs above, two modules needed one retry each: oidcc-ensure-registered-redirect-uri (Hybrid) and oidcc-codereuse (Form Post Hybrid). In two of five OidcFormPostTestSuite runs, one module (Form Post Hybrid, code token) hit the dropped connection in all 4 attempts and failed; it passed in the other runs.

How to run the tests, and the options

Prerequisites are a JDK, Docker or Podman, and a local Keycloak build (./mvnw install -DskipTests). The first run also pulls the conformance suite images from registry.gitlab.com.

Run a whole suite, as the CI matrix jobs do:

./mvnw package -f tests/conformance/pom.xml -Dtest=OidcTestSuite
./mvnw package -f tests/conformance/pom.xml -Dtest=OidcFormPostTestSuite
./mvnw package -f tests/conformance/pom.xml -Dtest=OidcDynamicTestSuite

Run a single plan:

./mvnw test -f tests/conformance/pom.xml -Dtest='org.keycloak.tests.conformance.oidc.OidcBasicTest'

The plan classes in the org.keycloak.tests.conformance.oidc package are:

• OidcConfigTest, OidcBasicTest, OidcImplicitTest and OidcHybridTest;
• OidcFormPostBasicTest, OidcFormPostImplicitTest and OidcFormPostHybridTest;
• OidcDynamicTest.

Option kc.test.conformance.retries (default 3): a module that does not finish with its expected result is run again, in a new plan, up to this number of times. The test fails only if all attempts fail, and then reports the result of the last attempt. Every retry is printed to the test output with the prefix [conformance retry], also when a later attempt passes. The option can be set as a system property, the KC_TEST_CONFORMANCE_RETRIES environment variable, or in keycloak-test.properties. Set it to 0 to turn retries off:

./mvnw package -f tests/conformance/pom.xml -Dtest=OidcTestSuite -Dkc.test.conformance.retries=0

When a module fails, its full conformance suite log is printed to the test output.

Proxy

Six of the Dynamic modules need the tunnel. In each, Keycloak fetches something the suite publishes:

• jwks_uri: oidcc-registration-jwks-uri and oidcc-refresh-token-rp-key-rotation.
• request_uri: oidcc-request-uri-unsigned and oidcc-request-uri-signed-rs256.
• sector_identifier_uri: oidcc-registration-sector-uri and oidcc-registration-sector-bad.

oidcc-ensure-request-object-with-redirect-uri sends its request object by value, so it doesn't need the tunnel. Here's a section to add to the issue, ready to paste; I'd put it right after "Executed tests and results":

Proxy for the Dynamic OP plan

In some modules of the Dynamic OP plan, Keycloak itself fetches a URL that the conformance suite publishes for the client it registered:

The suite publishes these URLs under its host name inside the containers' network (https://nginx:8443/...). Keycloak runs on the Docker host and cannot resolve that name, so without a proxy it cannot fetch the URLs and these modules cannot pass. oidcc-registration-sector-bad could even pass for the wrong reason: Keycloak would reject the registration because it cannot fetch the document, not because it checked the redirect URI.

The tests therefore start a small HTTP proxy in the test JVM (SuiteTunnel of the conformance-runner module). It accepts only HTTP CONNECT requests for the suite's internal host and port, and tunnels them to the port the suite's container publishes on the Docker host. The shared Keycloak server configuration of the OpenID provider tests (ConformanceServerConfig) points Keycloak's outgoing HTTP client at it, only for the suite's internal host name:

--spi-connections-http-client--default--proxy-mappings='nginx;http://127.0.0.1:<port of the tunnel>'

Requests to any other host are not affected. TLS stays end to end between Keycloak and the suite: Keycloak trusts the suite's self-signed TLS certificate through a truststore the tests write. Because that certificate does not cover the host name, hostname verification is turned off (tls-hostname-verifier=ANY), which is acceptable for a test server only.

The proxy needs no setup and has no option: it starts with the conformance suite containers and is always used. It does not affect the plans in which Keycloak does not fetch suite URLs (Config, Basic, Implicit, Hybrid and Form Post).

Implementation notes

• ConformanceServerConfig: Keycloak server configuration shared by the OpenID provider conformance tests
• conformance-runner: ConformanceTls holds the BCP195 TLS settings of the test servers, now also used by the OID4VCI tests; a browser interaction snapshotting the second login (prompt=login, max_age=1); the login page match covers the authorization endpoint
• AbstractConformanceTest: log the full logs of a failed module formatted upfront, as the test framework log filter drops the parameters
• a script based protocol mapper of this module (src/main/resources, with the scripts feature enabled) removes from the ID token the typ claim Keycloak always adds and, unless no access token is issued, the email claims, and the claims parameter mapper returns the essential claims requested with the claims parameter, so the modules warning about these finish with PASSED
• the suite's browser does not process the CSS of the Keycloak pages (SuiteOptions, browsercontrol_css_enable), where most of the time went
• modules finishing with REVIEW are documented next to their lists
• AbstractConformanceTest: a module not finishing as expected is run again, in a new plan, up to 3 times (kc.test.conformance.retries option), as modules occasionally fail for reasons outside of Keycloak, e.g. network errors or the timing of the suite's waits; every retry is printed to the test output, also when a later attempt passes

@lhanusov Hello, I sent the PR for OIDC conformance test as we discussed in the meeting.
If you have spare time, I would appreciate it if you could review the PR.

After that, I will work on OIDC-logout profile, FAPI 2.0 SP, FAPI 2.0 MS and FAPI-CIBA conformance test run by CI consecutively.

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.broker.KcSamlFirstBrokerLoginTest#testResetPasswordDuringFirstBrokerFlowWithoutExistingUserAuthenticator

Keycloak CI - FIPS IT (non-strict)

jakarta.ws.rs.InternalServerErrorException: HTTP 500 Internal Server Error
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.handleErrorStatus(ClientInvocation.java:250)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.extractors.DefaultEntityExtractorFactory$3.extractEntity(DefaultEntityExtractorFactory.java:41)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:146)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:104)
...

Report flaky test

@tnorimat don't forget to sign-off the commit 👍

@lhanusov Hello, thank you for the review.

My question, do you plan to use it in additional oidc tests in the next PR? Otherwise move ConformanceServerConfig and ConformanceRealmUtil under oidc, since their current responsibilities are OIDC-specific, and remove the unused additionalTrustedCertificates() hook.

Yes, I plan to use both in the next PRs. The PRs for FAPI 2.0 Security Profile, FAPI 2.0 Message Signing and FAPI-CIBA run on the same Keycloak server configuration, so the server is not restarted between the OpenID provider suites, and they add their realm building blocks (client policies, signing keys) to ConformanceRealmUtil. That's why I kept both classes in the shared package. You are right that additionalTrustedCertificates() is not used in this PR, so I removed it. The FAPI-CIBA PR will add it back, together with its only user (trusting the CAs of the TLS client certificates).

Make retries opt-in for OIDC initially. Adding these tests shouldn’t silently change how existing VCI/VP failures are handled.

I see. Retries are now off by default (AbstractConformanceTest.defaultRetries() returns 0). Only the OpenID Connect tests opt in, with 3 retries (AbstractOidcConformanceTest). The VCI/VP tests keep their current failure handling, which a VP suite run with the change confirmed. The kc.test.conformance.retries option still overrides the default for all tests; the README notes that the VCI/VP tests use a single alias, so they are not protected against late callbacks if someone enables it for them.

SuiteTunnel: Keep it encapsulated and test socket shutdown/error handling. Starting it lazily would avoid an unused listener in VCI/VP runs.

Regarding lazy start, OpenIdConformanceServer now creates the tunnel only when a Keycloak server configuration first asks for the proxy mapping, i.e. only ConformanceServerConfig. The VCI/VP runs never open the listener, and the startup code no longer has to close it on failure.

Regarding encapsulation, the tunnel stays package-private and is only reachable through OpenIdConformanceServer.keycloakProxyMapping().

Now, SuiteTunnelTest covers:

• forwarding a response the target sends only after the client closed its output;
• rejecting a CONNECT request for another target;
• closing the client when the target resets the connection;
• closing both sockets when forwarding fails;
• close() closing a connection still sending its CONNECT request, and an established tunnel on both sides;
• bracketing IPv6 addresses in the proxy mapping.

I checked that the shutdown and error tests fail when the corresponding fix is removed.

Ensure cleanup succeeds before starting another attempt. Currently cancellation can fail and the retry still proceeds; five seconds without browser logs also doesn’t guarantee that the browser has stopped. Please add focused coverage for cancellation failure and delayed callbacks.

I confirmed no retry after a failed cleanup. If a run fails after its module was created, the runner stops the module (DELETE /api/runner/{id}) and waits until the suite reports it stopped. If that fails or times out, the run now fails with ModuleCleanupFailedException (cause: the run's failure; suppressed: the cleanup failure), and that exception is never retried.

Regarding delayed callback, I agree that a quiet period without browser logs proves nothing, so I removed it. Instead, each attempt now uses its own suite alias (keycloak, keycloak-retry1, ...). A late callback of an earlier attempt's browser, e.g. an authorization response, goes to that attempt's callback URL and cannot reach the module of a later attempt. The static clients register the callback URL of every attempt, and the dynamically registered clients get theirs from the suite.

Regarding the coverage, the retry loop moved from AbstractConformanceTest to ConformanceAttempts in the conformance runner, so it can be tested without a suite.

• ConformanceApiClientTest, against a fake suite API, covers the cleanup when the module stops, when it is not running any more, when it cannot be stopped (HTTP 500), when it never stops (timeout), and that a successful run is not stopped.
• ConformanceAttemptsTest covers: the aliases of the attempts; retries until the expected result; retries after an exception; rethrowing the last exception; no retry after ModuleCleanupFailedException; and a late callback of attempt 1, arriving after attempt 2 started, reaching only attempt 1's alias.

I also checked a real retry against the suite: the second attempt ran under keycloak-retry1 and passed.

Keep the token-rewriting script clearly identified as a conformance configuration adjustment, with the reason for each change documented.

I renamed the script to conformance-adjustment-oidc-id-token.js.

• Header: it states that the script is a conformance test adjustment, used only by the realms of the OpenID Connect conformance tests, and not a recommended Keycloak setup.
• Removing typ: Keycloak always adds it; OpenID Connect Core does not define it, and the suite warns about it in oidcc-server.
• Removing the email claims: they go only when an access token is issued and the client did not request them in the ID token with the claims parameter. OpenID Connect Core 1.0: section 5.4 returns them from the userinfo endpoint in that case, and the suite warns in oidcc-scope-email and oidcc-alternate-happy-flow; OpenID Connect Core 1.0: section 5.5 covers the claims parameter exception.
• Names: the mapper registration (keycloak-scripts.json) and the realm mappers are named "conformance test adjustment".
• Deployment: regarding the line you linked, ConformanceServerConfig and the README now state that the scripts feature and the deployment of this module as a provider are only there for this adjustment.