Proposal of handling of module status

[barchw]

Description

Changes proposed in this pull request:

• Proposal of handling of module status

Pre-Merge Checklist

• As a PR reviewer, verify code coverage and evaluate if it is acceptable.

Related issues

[Ressetkk]

I think ADR should be focused around one solution that is already the one that we want to implement, rather than presenting possible solutions. That creates open questions which are in contradiction with Architecture Decision Record, because the architecture decision is yet to be made. I think we should analyse each of the solutions, present results, decide which solution to accept, and then prepare a decision record.

Moreover, I think we should really think again how our operator handles resources. Maybe it'd be better to just reduce the dependency in APIGateway CR controller to actually handle the lifecycle of application controllers, and maybe move some of the logic to separate controller...

Moving back to the topic - usually the status keeps the finite state of a resource and changes if the said state changes. That means setting a status rather should be a final step in resource reconciliation. Keeping Processing as transient state will make a problem with resource conflicts, if you for example set this state at the beginning of reconciliation, then you have to Get() resource again to fetch its updated state.

Ideally this should look like that:

1. User applies a CR
2. Controller schedules bunch of stuff
3. Controller sets up Processing status, finishes reconciliation and queues the next run in seconds
4. In the next run controller checks the state of each of the stuff we scheduled
5. If they are working/done, controller sets state to Ready and finishes work
6. If they failed, controller sets state to Error/Warning and finishes work - user attention is needed
7. If they still run, controller leaves Processing status and queues next run.

[mluk-sap]

Let's rework the document a little bit, so the proposed solution is described in details in the main section (like 'Decision') and put other ones in 'considered alternative solutions'.