Skip to content

Extract legacy store to separate maven modules - #11933

Merged
hmlnarik merged 26 commits into
keycloak:mainfrom
hmlnarik:10279-extract-legacy-to-maven-module
Jun 21, 2022
Merged

hmlnarik merged 26 commits into
keycloak:mainfrom
hmlnarik:10279-extract-legacy-to-maven-module

Conversation

@hmlnarik

@hmlnarik hmlnarik commented May 11, 2022 •

Copy link
Copy Markdown
Contributor

Description

In preparation for the Storage.X map storage functionality, modules and APIs have been consolidated so that the service layer won't differentiate between cached vs. non-cached, and federated vs. local storage, and legacy storage code is placed into its own modules.

Disclaimer: This PR is big. This is consequence of storage code being spread across many places - moving one class led to moving another etc. The commits are structured in a way that should help reviewing this PR, and it is the reason for intentional breaking the rule of "one commit per PR". The commits will be squashed upon merge.

Modules

Three new modules have been introduced, and data-oriented code from server-spi, server-spi-private, and services has been moved there:

  • model-legacy - contains all public facing APIs from legacy store, like User Storage API
  • model-legacy-private - contains all implementation that relates to user storage management (e.g. storage *Manager classes.
  • model-legacy-services - contains all REST endpoints that directly operate on legacy store, and have no meaning in the new store

These modules will be available as long as legacy store would be supported. After that period, they will be removed.

API impact

  • KeycloakSession has been simplified. Several methods have been deprecated in KeycloakSession. Please see Deprecated old KeycloakSession APIs commit

    • For KeycloakSession, all methods ending in localStorage and storageManager are deprecated. Instead of calling session.userStorageManager(), call session.users() instead. When code continues to call the deprecated methods, it will be redirected to the new methods. As an example, userLocalStorage() will now return the same result as users(), which might involve a cache if that has been enabled in the local setup.
    • Please review your code if it would depend on any functionality in regard to cached vs. noncached or local vs. federated storage.
    • No calls from Keycloak’s legacy code will any longer call deprecated methods; they will either call the new methods, or the functionality has been moved directly into the legacy modules. This way the deprecated APIs for example in KeycloakSession can be removed in the future without changing any Keycloak code.
  • KeycloakSession.userCredentialManager() is deprecated, user credential API has moved to UserModel.credentialManager(). Credentials for users previously managed via session.userCredentialManager().method(realm, user, ...) are now managed with user.credentialManager().method(...) which gets this functionality closer to API of users, and won't require a distinction of where the user data is physically located.

  • New private SPI org.keycloak.services.resources.admin.ext.AdminRealmResourceSpi for registering endpoints on the realm admin endpoint. Some of the realm endpoints in services were tightly coupled with user storage or LDAP. To avoid the services to depend on the storage and the legacy modules, this SPI has been introduced.

Access to the legacy store

While some APIs have been deprecated and classes moved to other modules, there will still be a period when legacy store will be available in Keycloak. Caution was executed to break this compatibility minimally, and the code using legacy store only needs to depend (directly or indirectly) on model-map-legacy module in addition to server-spi.

For example, the existing quickstart example still compiles. Still, those who wrote Keycloak extensions based on the Keycloak user-storage-jpa Quickstart (https://github.com/keycloak/keycloak-quickstarts/tree/main/user-storage-jpa) should review the migration guide to find out about the changed semantics in the deprecated methods especially on KeycloakSession.

Export / import

  • New storage only supports the latest export file representation, it performs no migration

  • Parts of the import/export functionality has been moved to the legacy module as it depends on the user storage APIs. Work will continue to refactor this so that the service layer will forward a file or InputStream to the storage layer without parsing the JSON to a representation. This will remove the need to have fields from older Keycloak versions in the REST representations in the service layer.

Out of scope of this PR / Future work

Completing the change of the export / import flow so that the datastore can directly perform updates rather than converting representation to model and back to datastore representation. This will allow sharing no-downtime capabilities of a store with that of export / import functionality, e.g. by leveraging future YAML static store directly as an export format.

Fixes: #10279
Fixes: #10281

@hmlnarik
hmlnarik force-pushed the 10279-extract-legacy-to-maven-module branch from f89a56f to 587225b Compare May 11, 2022 06:16
@hmlnarik hmlnarik self-assigned this May 12, 2022
@hmlnarik hmlnarik added area/storage Storage layer — JPA entities, database schemas, Liquibase migrations, caching team/storage-sig labels May 12, 2022
@hmlnarik
hmlnarik force-pushed the 10279-extract-legacy-to-maven-module branch 9 times, most recently from f007c19 to 5d48f34 Compare May 30, 2022 08:02
@hmlnarik
hmlnarik marked this pull request as ready for review May 30, 2022 10:29
@hmlnarik hmlnarik added area/core Core server — startup, configuration, providers/SPIs, cross-cutting concerns area/quarkus labels May 30, 2022
@pedroigor

Copy link
Copy Markdown
Contributor

@hmlnarik Isn't keycloak-model-infinispan also legacy? Does it make sense to have everything legacy-related within the same module structure?

@pedroigor pedroigor left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hmlnarik @ahus1 A few comments, but overall LGTM. Level hard kind of change :)

import org.keycloak.services.resources.admin.ext.AdminRealmResourceProviderFactory;
import org.keycloak.services.resources.admin.permissions.AdminPermissionEvaluator;

public class LdapServerCapabilitiesRealmAdminProvider implements AdminRealmResourceProviderFactory, AdminRealmResourceProvider {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the new AdminRealmResourceProviderFactory indicates a requirement that should be available from the new store? In this case, the possibility to expose information about the store itself as well as perform operations on it (e.g.: test connection).

Is federation/ldap going to be refactored or replaced by the new store in the future? How the AdminRealmResourceProviderFactory is aligned with the new store?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pedroigor - federation and ldap will be different with the new store; they would kind of "blend in" like any other map storage. In order to do that, a store can handle credentials and authentication. This is already visible in the refactoring for credentials.

For LDAP, you can validate and change a password for a specific user.

For Kerberos, you'd send down the SPNEGO token to receive a reply how to continue with the authentication or which user it has been mapped to.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the new AdminRealmResourceProviderFactory indicates a requirement that should be available from the new store? In this case, the possibility to expose information about the store itself as well as perform operations on it (e.g.: test connection).

Yes, this is the case. The store-specific endpoints would implement this interface and would be used by e.g. admin UI, and example could be a reconfiguration of storages or running a migration for a particular store, or testing a password for a particular store configuration just like we had it for LDAP in the legacy store.

This change is broader at the same time: just as storage, any other (optional) feature may now add an admin endpoint for itself inside realm admin REST endpoint which would only be available if the feature is enabled.

@mhajas mhajas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hmlnarik @ahus1 Thank you for the PR. Overall, the changes look good, I like the way this is heading.

However, the PR is huge, it is really hard to see what changes are just moving some code from one module to another, and what was added/changed. I think I need another review round and even then I am not sure I would be confident to say there are no issues.

I understand that in this state it is really hard to split the PR, but if you see any option of doing that, I would vote for that.

Anyway, see my comments in the code.

@hmlnarik

hmlnarik commented Jun 8, 2022

Copy link
Copy Markdown
Contributor Author

@pedroigor Thank you for the review! Kind of level hard for reviewers as well, I admit :-)

@hmlnarik Isn't keycloak-model-infinispan also legacy? Does it make sense to have everything legacy-related within the same module structure?

It is, both model-infinispan and model-jpa are legacy. Either way would work for me - keeping it as is, or restructuring it - and I have no strong arguments for either. I kept the directory structure untouched in this regard since the modules will eventually disappear anyway, and the map and legacy modules coexist today as well. If the arguments prevail for the latter, would it be possible to do it separately after merging this PR?

@stianst stianst left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No objections from me on this one. Although I have NOT reviewed it thoroughly, but happy with the general approach.

Would be great to get a PR to release notes and migration guide prior to merging this one though. One thing I'd like to understand better is how much of a breaking change this is to folks with custom providers.

@hmlnarik
hmlnarik force-pushed the 10279-extract-legacy-to-maven-module branch from b78a39c to e1e2467 Compare June 17, 2022 08:30
@stianst

stianst commented Jun 17, 2022

Copy link
Copy Markdown
Contributor

LGTM and will approved after open discussion are concluded, and keycloak/keycloak-documentation#1541 has been reviewed and approved by others as well.

@hmlnarik

Copy link
Copy Markdown
Contributor Author

Clustering tests have failed because of #12359, tracked in #12560

@mposolda mposolda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hmlnarik @ahus1 Approving. Kudos for all your work on this!

@mhajas mhajas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a big step for the new store, thank you @ahus1 @hmlnarik!

@vramik vramik left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR LGTM. Although I didn't do a deep review but I like the direction where it's going. Thank you @hmlnarik @ahus1

@hmlnarik
hmlnarik merged commit ae7c01b into keycloak:main Jun 21, 2022
@hmlnarik

Copy link
Copy Markdown
Contributor Author

Thank you everyone for your reviews and help!

@tsaarni

tsaarni commented Jun 30, 2022

Copy link
Copy Markdown
Contributor

@hmlnarik I think there was a small typo in this PR that now causes syntax error when compiling main with Eclipse compiler.

See #12837 and tiny PR to fix it #12838

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/core Core server — startup, configuration, providers/SPIs, cross-cutting concerns area/dist/quarkus Quarkus distribution — packaging, container images, server config (kc.sh) area/storage Storage layer — JPA entities, database schemas, Liquibase migrations, caching

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

Update DefaultKeycloakSession Extract legacy store code into maven module