Build system cleanup/revision

[bertfrees]

The build system has become a real obstacle, especially for external contributors. There are various things we can do to make it more manageable.

February 2017

• [done] The super project: Create aggregator projects #34
• [ready] Put all runtime dependencies in POMs to make testing less painful: Add OSGi runtime dependencies to all POMs #23, Dependencies clean up pipeline#503
• [ready] Minimize number of places where dependency versions are declared: Dependencies clean up pipeline#503
• Fix circular dependencies: Fix circular dependency between fileset-utils and mediatype-utils pipeline-modules-common#95, Fix circular dependency between fileset-utils and mediatype-utils pipeline-mod-braille#144, Fix circular dependency between fileset-utils and mediatype-utils pipeline-scripts#100
• Use same artifacts for compiling and running: Dependencies clean up pipeline#503
• Replace commons-logging dependencies with jcl-over-slf4j: Dependencies clean up pipeline#503
• Remove SLF4J implementations (notably logback-classic) from the (runtime) dependencies in order to make it easier to control logging in tests: Dependencies clean up pipeline#503

May, June, July 2017

• Generalize and refine super build script (more "Make-based" and less "Maven-based"): daisy/pipeline@64181df105
• Write tools to facilitate working with super project (git subrepo-status): daisy/pipeline@6b18d6f390
• Further automate release process: Make release process more automatic #98

May 2018

• Stop using remote snapshots repositories: Stop using remote snapshots repositories (notably sonatype-nexus-snapshots in daisy-parent) #75
• Include only one version of guava (not a different one specially for epubcheck): Fix issue of having to include multiple versions of Guava pipeline-assembly#132
• Change the scope of the slf4j-api dependency to compile (currently provided in many places)
• Make build work with -f <path-to-pom>: e.g. Make build work with -f <path-to-pom> pipeline-assembly#117

2020

• Instead of having dependency <exclusions> in the BOMs, exclude them from the declared dependencies in the artifact POMs (or make provided).

---

Other

• [ready] Felix annotations: Use Felix SCR annotations for generating the scr:component XML files #22
• epubcheck-adapter: why not use parent?
• Move lifecycleMappingMetadata of scripts-parent, scripts-utils-parent, etc. to modules-parent
• Include only one version of Jing
• Fix warnings: Fix all build warnings #42
• Make offline building possible
  • including integration tests with maven-invoker-plugin
  • daisy/pipeline@5ef0f40ec8
  • use common-entities in tests that depend on certain resources
• Use the same dependency versions everywhere so that there can be no confusion about which version Maven includes when running tests. (see make check-versions)
• [doing] Full build (no internal versions): Support full build #40
• [doing] Assembly clean up: POM clean up pipeline-assembly#75
• Generate Require-Bundle metadata: Generate Require-Bundle headers from source files instead of nextCatalog trick #41

See branches:

• daisy/pipeline@master...dependencies
• assembly-modularize (local)
• daisy/pipeline-assembly@master...pom-clean
• daisy/pipeline@master...versionless
• offline (local)

[josteinaj]

Possibly related to #41: To enable xprocspec testing we currently have to add a Java file under src/test/ and list the same dependencies as are listed in the POM. It's basically something you copy from another project and edit the list of dependencies. Is it possible to remove this requirement, and just enable xprocspec testing for all module projects (unless explicitly disabled in the project pom)?

[bertfrees]

In daisy/pipeline@dependencies...script-tests I've done some work to make it easier to create tests. The boilerplate in both Java and POM is reduced to a minimum this way. I hope this is a good enough improvement so that we don't have to make it completely automatic which would be a step too far IMO. That's just too magic for me. It's also not easy to do, technically.

[bertfrees]

@rdeltour I suggest we start with these ones (that I put at the top). Does that sound OK to you?

[bertfrees]

See PR daisy/pipeline#503.

[bertfrees]

• Fix circular dependencies

The only circular dependency that I know of at the moment is between fileset-utils and mediatype-utils. (px:unzip-fileset uses px:mediatype-detect which uses px:fileset-load). We can solve this by replacing all px:unzip-fileset with the new px:fileset-unzip (daisy/pipeline-modules-common@0dcbd4b). px:unzip-fileset is used in epub3-to-daisy202 and epub3-to-pef.

[bertfrees]

@josteinaj Could you please review daisy/pipeline@script-tests...remove-unzip-fileset? The tests succeed.

[josteinaj]

@bertfrees looks good

[bertfrees]

I've moved the items that haven't been done to "Later" and cleared the milestone for now.

[bertfrees]

• Stop using remote snapshots repositories: Stop using remote snapshots repositories (notably sonatype-nexus-snapshots in daisy-parent) #75

@rdeltour I'd like to do this one. Any objections?

[bertfrees]

@josteinaj It seems that unzip-fileset was also used in epub3-dtbook-migrator. Have you noticed that?

[rdeltour]

Stop using remote snapshots repositories: #75

@rdeltour I'd like to do this one. Any objections?

No objections, go ahead.

[josteinaj]

@bertfrees yes, I'm trying to fix the nordic migrator (only some evasive weird issue with epubcheck remaining) (nlbdev/pipeline@fix-nordic-migrator), and one issue was the renaming of unzip-fileset. It wasn't a big deal though, I just had to rename it: nlbdev/pipeline@bce4267

[bertfrees]

OK. Just wanted to check because I also just come across this issue in the sbs branch.

[bertfrees]

See daisy/pipeline-framework#149