Refinement of survey responses - code organisation

[bertfrees]

We are currently in the process of analysing your responses to the Pipeline usage survey held earlier this year. Some of the feedback requires some more elaboration or discussion. Below are some responses in the topic "code organisation":

---

• Q: Please describe any functionality you would like to see in Pipeline in future
  • One organisation suggested to extract parts of the code into standalone libraries:

    It could be nice some things were separated into separate and independent projects. For instance, the actual braille production and TTS production could be separate tools, while Pipeline 2 could be a more pure HTML transformation (i.e. XSLT) tool.

---

@josteinaj already confirmed that it would not be required to develop separate user interfaces for these libraries. It is enough to have an API.

I would like to know:

• Because the libraries will naturally be based on the existing code, the language will be Java. Is that a limitation?
• One of the goals is that such standalone libraries would attract more users and developers. It would be beneficial for everyone if the maintenance of these libraries could be a joint effort between DAISY and other partners. For instance, the Dotify library is maintained by TexTalk (for MTM), Dedicon and myself, and we all review each others contributions. How can we create interest for helping to maintain the libraries?

[josteinaj]

I think Java is fine.

Removing XProc would be ideal. Less XSLT can also be an advantage, depending on what it is replaced with.

The most important thing is that the code is easy to understand and maintain for new developers.

And that snapshot versions (i.e. :latest for Docker images) can be used to test in real production environments, independent of the main Pipeline 2 release cycle. This will make it easier to iterate on the code in periods of active development.

Being able to integrate into agency-specific workflows and systems is also important. So the agencies that want to use (and hopefully help maintain) these separate tools, should also help with defining the API. That will hopefully reduce the amount of glue code that is necessary.

In addition to an API, I think some basic CLI is also useful.

[josteinaj]

What do you think of this setup?

• It would make it possible to have organization-specific scripts
• Scripts can be written in any language as long as they use the orchestrator API
• Easy to share scripts between organizations that use Pipeline 2
• AMQP would integrate better with architectures that are based around a message bus
• When separating some code into separate tools (for instance the braille scripts), it could use the orchestrator API, and thus be more indepenent of the technologies in the main Pipeline 2

[egli]

What do you think of this setup?

Doesn't that make thing more complicated by introducing

1. yet another abstraction layer (between the clients and the Pipeline
2. another communication protocol besides REST

Why do organization specific scripts have to be behind the same API as the Pipeline scripts?

[josteinaj]

I think organization specific scripts behind the Pipeline 2 API could help involve more developers and encourage sharing of production steps between organizations. But I see your points.

[bertfrees]

One obvious benefit I can think of is that the same user interface could be used for different types of scripts.

If, in your diagram, "Pipeline 2 orchestrator API" is simply replaced by "Pipeline 2 HTTP API", your idea is not so far from my idea of writing a script/job API backend that dispatches jobs to one or more Pipeline web servers.

AMQP would integrate better with architectures that are based around a message bus

Is this an actual use case?

[josteinaj]

I'm thinking of Nota, since they use a message bus. However, they don't use Pipeline 2. So it would have to be a discussion with them. If they're not interested, I don't see a use case for AMQP. I think Nota does a lot that could be useful for other organizations, but since we all have different architectures, it's not so straight forward to share production steps.

The nordic migrator is another example that could be integrated in Pipeline 2 with such a backend API, although that would probably just be HTTP and not AMQP.