Realm export normalization

[sonOfRa]

What this PR does / why we need it:
When trying to move an existing keycloak installation with no configuration management to keycloak-config-cli, it can be hard to figure out which parts of the realm export are actually needed to reproduce the state when re-importing the configuration.

This PR can serve as a (partial) solution to this problem, by taking a realm export from a certain version of Keycloak, comparing it against a reference export of a "barebones" Keycloak realm from the same version, and then creating a yml (or JSON) file that only contains changes that are actually required.

This PR will not be a complete solution that works for all cases, but it will supply an at least somewhat minimized view of clients and some related settings, which can then easily be merged with a full export in order to get a smaller view of the file. Support for additional parts (identity providers, authentication flows, etc) can be added in later steps to improve this export.

This PR includes a "mode switch" that allows users to switch between IMPORT and NORMALIZE modes using the property run.operation=NORMALIZE. IMPORT is the default, and is used when no option is supplied by the user.

This PR includes an upgrade to Spring Boot 2.7.5, which contains a fix for spring-projects/spring-boot#32559. The @DefaultValue annotation is used to move configuration from the default application.properties file directly to the code. This is needed so that only the needed configurations for the chosen mode (IMPORT or NORMALIZE) are activated.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #799

Special notes for your reviewer:
A custom properties application-normalize-dev.properties file for testing the export functionality was added. Activating this profile enables normalization mode, and expects an input realm to be present at the top-level exports directory

exports
├── in
│   └── realm.json
└── out
    └── demo1234.yaml

The file in/realm.json will be imported, and the file out/demo1234.yaml is created by the process, where the realm name of the input realm is demo1234

Run the application with the normalize-dev profile active to normalize a given realm.

The configuration parameter normalization.output-format can be used to switch between YAML (the default, if no option is given) and JSON output.

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[codecov]

Codecov Report

Attention: Patch coverage is 20.45728% with 661 lines in your changes missing coverage. Please review.

Project coverage is 83.08%. Comparing base (dbb8950) to head (93c5fbc).
Report is 219 commits behind head on main.

❗ Current head 93c5fbc differs from pull request most recent head fe9de52

Please upload reports for the commit fe9de52 to get more accurate results.

Files	Patch %	Lines
...ycloak/config/provider/KeycloakExportProvider.java	7.69%	84 Missing ⚠️
...ervice/normalize/AuthFlowNormalizationService.java	6.94%	67 Missing ⚠️
...g/service/normalize/GroupNormalizationService.java	8.33%	66 Missing ⚠️
...ig/service/normalize/RoleNormalizationService.java	7.46%	62 Missing ⚠️
...g/service/normalize/RealmNormalizationService.java	24.28%	53 Missing ⚠️
...ormalize/IdentityProviderNormalizationService.java	8.92%	51 Missing ⚠️
.../service/normalize/ClientNormalizationService.java	12.72%	48 Missing ⚠️
...ce/normalize/ScopeMappingNormalizationService.java	8.51%	43 Missing ⚠️
...ice/normalize/ClientScopeNormalizationService.java	11.36%	39 Missing ⚠️
...loak/config/KeycloakConfigNormalizationRunner.java	20.93%	34 Missing ⚠️
... and 10 more

Additional details and impacted files

@@              Coverage Diff              @@
##               main     #818       +/-   ##
=============================================
- Coverage     95.46%   83.08%   -12.39%     
- Complexity     1302     1339       +37     
=============================================
  Files            76       96       +20     
  Lines          4213     5037      +824     
  Branches        467      621      +154     
=============================================
+ Hits           4022     4185      +163     
- Misses           94      755      +661     
  Partials         97       97               

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[manuschillerdev]

Would it be possible via some kind of diffing to keep params for fields with env var substitutions?

Example:

{
  "realm": "my-realm",
  "smtpServer": {
    "replyToDisplayName": "$(env:SMTP_REPLY_TO_DISPLAYNAME)",
    "port": "$(env:SMTP_PORT)",
    "host": "$(env:SMTP_HOST)",
    "replyTo": "$(env:SMTP_REPLY_TO)",
    "from": "$(env:SMTP_FROM)",
    "fromDisplayName": "$(env:SMTP_FROM_DISPLAY_NAME)",
    "envelopeFrom": "$(env:SMTP_ENVELOPE_FROM)"
  },

When I import this realm, make changes and export the realm again I'd like to keep the params for SMTP instead of having to replace the actual values from the export with the parameters again.

I have no idea what that means regarding the implementation. Just as an idea to make working with the huge JSON-Files easier :)

[sonOfRa]

@manuschillerdev I think that might be a valuable future extension of this feature!

In theory, one could provide a map of "reverse injections" where you specify something like "When the field realm.smtpServer.host has the same value as the resolved environment variable "SMTP_HOST", then replace the value with $(env:SMTP_HOST). This seems to me like a valuable addition to this feature in order to make the process of going from "I have a keycloak instance that has been configured via admin-ui" to "I have a relatively minimal yaml file that represents my config state"

However, I think I'd like to get the base idea of this PR (minimize a realm.json acquired from a full Keycloak realm export) done first, and then build additional features on top of that

[weand]

i tried this PR on KC20.0.5.

After adding baseline files for 20.0.5 and building the cli, following error occurs when running it against our existing realm. Seems like there are remove(...) calls on an immutable map...

Caused by: java.lang.UnsupportedOperationException: null
        at java.base/java.util.ImmutableCollections.uoe(ImmutableCollections.java:142)
        at java.base/java.util.ImmutableCollections$AbstractImmutableMap.remove(ImmutableCollections.java:1075)
        at de.adorsys.keycloak.config.service.normalize.RoleNormalizationService.normalizeClientRoles(RoleNormalizationService.java:83)
        at de.adorsys.keycloak.config.service.normalize.RoleNormalizationService.normalizeRoles(RoleNormalizationService.java:57)
        at de.adorsys.keycloak.config.service.normalize.RealmNormalizationService.normalizeRealm(RealmNormalizationService.java:150)
        at de.adorsys.keycloak.config.KeycloakConfigNormalizationRunner.run(KeycloakConfigNormalizationRunner.java:92)

[sonOfRa]

Hi @weand!

Can you provide a realm export for the realm you tried this on? Note that realm exports will contain sensitive data like the client secrets and all your realm keys (for signing JWTs). If it's not possible to censor the realm enough for it to be exportable to a public place, could you maybe find a certain configuration of roles that triggers this bug? I'll also take a look in the meantime if I can find what causes this.

[weand]

@sonOfRa Check this reproducer export realm-export-uoe.zip

It's as simple as this: Simply disable one of the default clients, e.g. realm-management and security-admin-console in my example.

BTW: I could not figure out how to enable debug output (which prints the root exception). With that:

java -Dspring.profiles.active=debug \
  -jar $script_dir/keycloak-config-cli-20.0.5.jar \
  --normalization.files.inputLocations=$script_dir/normalize/in/*.json \
  --normalization.files.outputDirectory=$script_dir/normalize/out \
  --run.operation=NORMALIZE
  --debug

Log looks like this:

2023-03-14 15:09:13.382  INFO 2360748 --- [           main] d.a.k.c.s.n.RealmNormalizationService    :
 Exporting realm foo-bar
2023-03-14 15:09:13.486 ERROR 2360748 --- [           main] .a.k.c.KeycloakConfigNormalizationRunner :
 null

To see the root exception I had to remove that condition

Any advice on how to run with debug output enabled would be great.

[sonOfRa]

With the debug profile, there should be debug logging enabled, I'll try to see why that isn't showing up there for those messages.

I see that in the export you provided, there are no roles present whatsoever. Did you, by chance export the realm via the admin console's export functionality? For reasons I've never understood, that export is incomplete and doesn't include everything.

The assumption for this tool is working on a full export, as acquired via one of the methods listed here: https://www.keycloak.org/server/importExport#_exporting_a_specific_realm

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
24 Code Smells

No Coverage information
0.5% Duplication

[francis-pouatcha]

@jonasvoelcker what do you think. Can we look deeper into this feature?

[jonasvoelcker]

Hi @francis-pouatcha,

as mentioned in one of the other merge requests I am currently not available for any work on community things. Our project requires other things and additionally I am about to get a second child in a few days so I can't invest any freetime 😉

Best Regards
Jonas

[sonOfRa]

I currently cannot spare time to work on this PR further; I switched to another employer and we're not using the feature here. But if any of the people trying to use the feature are willing to take over development on this PR, feel free to do so; I place the changes I've made here in the public domain and anyone is free to use them to continue development of this feature with my changes as a base

[srose]

Maybe talk about the topic in Vienna tomorrow?

[thomasdarimont]

I just gave this a spin and aligned the changes from this branch with the latest keycloak-config-cli version in main
https://github.com/thomasdarimont/keycloak-config-cli/tree/feature/realm-normalization
I integrated all commits from this PR with one merge commit with proper config resolutions. I had to do make some changes to adapt the code for the newer Spring Boot version 3.x.
See: https://github.com/adorsys/keycloak-config-cli/compare/main...thomasdarimont:keycloak-config-cli:feature/realm-normalization?expand=1

I did only minimal changes to get it working again. I think this needs some refactoring (config property sharing, use profiles for conditional beans instead of propertys, revise naming, etc.) and I think we could (and should) extend this to support a "realm export" from an "live" realm.

Note that the normalization does not support some features of KC24/25, e.g. organizations etc.

The build works again I could successfully "normalize" a realm export generated from KC 25.0.5.
See: https://gist.github.com/thomasdarimont/7c434fe6dcf20fa28a06cc38e15d6105

[francis-pouatcha]

@thomasdarimont would you send a pull request to sonOfRa:feature/export-diff-yaml or to adorsys:main?