Repository navigation
[KEYCLOAK-16780] - Allow batching writes to storage when running migration - #7717
Conversation
hmlnarik
left a comment
There was a problem hiding this comment.
Thanks @pedroigor , looks good to me. This seems a good idea and I like the fact that this is limited to the migration. Have you tried any different limit, e.g. 500?
|
Also, has the pipeline been run on this PR? Since it touches migration, I'd rather have its results prior to merging. |
Co-authored-by: Hynek Mlnařík <hmlnarik@users.noreply.github.com>
d7fae73 to
e83c98d
Compare
|
@pedroigor Thanks! Will be good to have pipeline results for migration tests as @hmlnarik mentioned. Also I wonder that we may still reach the limits during migration for example with 5000 realms? I wonder that we may also need to increase the transaction timeout for the migration transaction? This can be done with KeycloakModelUtils.runJobInTransactionWithTimeout . But maybe can be done as a follow-up (if needed), so I don't have anything against merge this PR as is. |
vramik
left a comment
There was a problem hiding this comment.
Thank you @pedroigor for the PR. Great work, I have just one question.
| private PersistenceExceptionConverter(EntityManager em) { | ||
| private PersistenceExceptionConverter(KeycloakSession session, EntityManager em) { | ||
| batchEnabled = session.getAttributeOrDefault(Constants.STORAGE_BATCH_ENABLED, false); | ||
| batchSize = session.getAttributeOrDefault(Constants.STORAGE_BATCH_SIZE, 100); |
There was a problem hiding this comment.
Do we have some performance results with different STORAGE_BATCH_SIZE values? I think we should set the default value based on best results, wdyt?
There was a problem hiding this comment.
Sure. That number is what worked well for the use case we are testing using 1k realms.
Personally, I don't think it is critical for this PR. We can always change the value and users are free to set any value they want.
There is more work in the backlog that could help with this. But they are a follow-up to this PR and the other one related to multi-realm.
It works very well for our current scenario using 1k realms. I won't be surprised if running 5k the results are also good. This is already an improvement. So I think any other could be done as a follow up. |
|
Thanks @pedroigor, let's wait for the pipeline results. If those are ok, changes looks good to me. |
|
@pedroigor +1 to do as a follow-up if needed. We have https://issues.redhat.com/browse/KEYCLOAK-16615 for that. I will approve once you have this migration test passing in the pipeline. |
hmlnarik
left a comment
There was a problem hiding this comment.
LGTM, thanks @pedroigor ! The pipeline has revealed no regressions.
Please squash the two commits before merging.
|
@hmlnarik Going to send changes to documentation. |
These changes aim to:
Expected result:
555528ms, whereas with these changes migration happens in54595ms.During migration, we should expect degradation due to the number of changes being made. By using a batch approach (as recommended by Hibernate, btw), we can improve performance and startup.
We should expect better results if running using #7712.