Repackaging of egeria distribution & dependency streamlining

[planetf1]

We have made a number of changes to reduce unnecessary dependency chains & ensure Egeria can more easily be actually run in the flexible ways the architecture supports.

Very recent changes have included ensuring the connection factory loads connectors dynamically, rather than having hard dependencies (maven), as well as offering an optional profile for building the server chassis without all OMASs etc in place, rather depending on runtime scanning of classes. We also have a growing number of connectors being developed externally which need to be easy to build and deploy

Currently the main egeria distribution

• uses our 'full' server chassis
• packages many connectors wil all dependencies

This issue is opened to streamline this such that we:

• use our 'thin' server chassis
• package connectors in a thinner form (without egeria dependencies)

The end result should be that the assembly can be used as today.

We will also need to handle side effects from this change, such as how developers run and test code, the execution of FVTs, changes to container deployment, error handling, documentation, whilst also working within restrictions of various tools such as maven

Benefits include

• More flexible deployments matched with architecture
• Potential Lower security attack service
• Quicker & smaller build

Current list of potential tasks:

• Add all dependencies for servers & clients into distribution (harmless, enabler)

• Repackage connectors without dependencies (harmless, enabler)

• Reapackage clients/samples without dependencies - update readme/docs as needed

• Use skinny version of server chassis which will also require

  • Ensure FVTs work
  • Check impact on developers
  • Ensure container deployment works

• Cleanup - Review samples/utilities (In distribution only where useful)

• Cleanup - Check dependency chains

• Update docs on writing connectors

• Full Run through usual pre-release tests (charts, samples etc)

• Test a custom connector works ok

• Ensure changes work through maven (likely 1tst) and gradle (to follow)

Open related issues:

#1607 Rethink ConnectorConfigurationFactory < much of this is now likely addressed
#4455 investigate use/recommendation of 'provided' scope for connectors - this is a technical aid in some areas where a compile time dependency is needed ie for a connector, but where we know the runtime is present
#4280 Distribution open connector archives appears in both samples & utilities < just a cleanup
#3370 Gradle Build prototype < will continue this work. Currently need to get FVTs working
#2671 Cassandra Dependency < not tightly dependent, but somewhat related

cc: @mandy-chessell @lpalashevski @bogdan-sava

[planetf1]

@lpalashevski Are you currently working on any aspects in this area? Or do you have any additional input or related issues to link to?

[lpalashevski]

Not at the momenet. I remember Bogdan was still testing some aspects with maven depencenes and the slim build for server chassis I believe its related to 4455. Lets align on the call today and plan how we continue/devide work on specific areas.

[planetf1]

@bogdan-sava assigning this to you as I believe you were running with this.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[bogdan-sava]

The list of egeria connectors:

1. • audit-log-console-connector
2. • audit-log-event-topic-connector
3. • audit-log-file-connector
4. • audit-log-slf4j-connector
5. • avro-file-connector
6. • basic-file-connector
7. • cassandra-metadata-extractor-connector
8. • cohort-registry-file-store-connector
9. • configuration-encrypted-file-store-connector
10. • configuration-file-store-connector
11. • csv-file-connector
12. • data-engine-proxy-connector
13. • data-folder-connector
14. • elasticsearch-integration-connector
15. • graph-repository-connector
16. • inmemory-open-metadata-topic-connector
17. • inmemory-repository-connector
18. • janus-connector
19. • kafka-integration-connector
20. • kafka-open-metadata-topic-connector
21. • omrs-rest-repository-connector
22. • open-lineage-janus-connector
23. • open-metadata-archive-file-connector
24. • openapi-integration-connector
25. • readonly-repository-connector
26. • security-manager-tag-connector
27. • security-officer-tag-connector
28. • spring-rest-client-connector

As a first step I'll remove dependency for elasticsearch-integration-connector and graph-repository-connector, and add jar-with-dependencies for them.
And place the jars with dependencies in ./lib of distribution

[planetf1]

Agree on making small incremental steps. I would suggest we don't add anything into 3.3 at this point, but target 3.4 after we branch next week. It will give time to ensure adequate review and informal testing.

The graph repo connector pretty much has non-egeria dependencies - though given the way the assembly works we will be duplicating them all into lib. This also needs doing in the gradle assembly. Should be Harmless except in size . I don't see it would cause issues with demos etc, but the docker image/helm charts should be tested as part of the change to ensure (specifically on the graph repo)

In the case of the elasticsearch connectoir we'll pull some libraries in too, which would benefit from the 'what is the chassis' / provided aspect of what is described above as this would remove the duplicated libraries

We also need to agree whether we'll go with copying into 'lib' of the distro or jar-with-dependencies in general. either should work when running the distribution but there are other pros/cons - we don't need to decide this right now and can go with the latter for now.

[planetf1]

A simplification - we don't need to over-analyze what the server chassis consists of for now, instead:

first, Connectors .. (and other modules that run on the server chassis) can be built as jar-with-dependencies (as in the example you gave of the janus connector) - and I would probably do this across all connectors

At this point our build will grow substantially, and there can be some confusion debuggin, - though should be functional. Probably a good idea for beginning of release cycle.

Then we can incrementally

• remove connectors from the server chassis itself (ensuring they are in the assembly) [ impact only on intellij etc]
• modify each connector to add server-chassis-spring=provided (maven+gradle), and remove unnecessary dependencies (this should shrink the lists a lot)

As we do this the build will shrink again.

Since we are always using 'server-chassis' as provided, that transitive dependency chain is always a true reflection of what is really provided for that release, so if something is removed that another module depends on, we'll automatically get a build error (symbol not found etc) & so be able to correct. ie it's adaptable.

If in future we can apply the same technique to OMASs, indeed for any modules that are not in the server chassis itself, but depend on that runtime and can assume it is there.

(We may need to add server-chassis-spring to the exclusion list for the dependency checker, or remote provided scope)