Graph generation

[jcendava]

Hi.

In the documentation, it suggests the SDK and/or some tooling can be used to generated more complex graphs. Is there any additional guidance on this?

Manifest files will be able to represent simple calculation manifest but to handle larger, more complex systems, we will have to write Graphs as code using our SDK.

Also, can graphs be modified at runtime, i.e. by the first (or another) pipeline task?

Thanks
James

[jawache]

Yes they absolutely can, we just added in some code to support that. It was quite a complex change, it's not merged into the main branch yet but you can find an example of a plugin which does that here. It's pretty undocumented at the momenet but you might be able to glean how it works from this code: https://github.com/Green-Software-Foundation/if/blob/functional-architecture/src/models/group-by.ts

Specifically we've added a metadata part that the plugin needs to return and currently it's set to "groupby" and it needs to return child nodes rather than outputs.

@jmcook1186 and @narekhovhannisyan I think for this type of plugin we need the metadata to be more about what it returns, currently the metadata says "groupby" but thats the name of the first plugin that uses this method of returning children rather than the kind of the plugin. Maybe the kind should be children since it returns children nodes? Thoughts?

EDIT - Sorry @jcendava I can see the original discussion was from Jan 5th and I just got notified because it was closed.

[jcendava]

Thanks @jawache, that's really interesting. If I understand correctly, the changes are to allow definition of models that return child-nodes rather that outputs, and the group-by model is an implementation of this, that returns child-nodes based on groupings of inputs.

Currently, we're trying to understand whether a single child pipeline could execute such a model, query a service and then return a number of children for further processing, based on the resources returned (details to follow on /hack discussions in due course).

If that's possible, how would the new children's pipelines be specified?

(fyi I'd closed this ticket as I posted a new one under the /hack Q&A, so I got two answers :-) )

[jawache]

So there are two ways to do this right now:

1 - Have your plugin return a flat array of everything and then use the group-by plugin to allow people to group them how they want. E.g. return a flat array of observations for every server for all regions and then use group by to group them under region > some-server-unique-id. The adv of that approach is that you are allowing others to group data however works for their use case and also so you don't have to implement any grouping on yourside.

tree:
  children:
    application:
      pipeline:
        - your-plugin
        - group-by
        - some-other-plugin

• your-plugin will run and return a flat array.
• group-by will run and group those into a new child structure.
• some-other-plugin will run only on those child nodes and the current node will be treated as a grouping node (no inputs)

2 - Create a plugin which queries you DS and returns components already grouped how you would like them to be represented in the tree.

tree:
  children:
    application:
      pipeline:
        - your-plugin
        - some-other-plugin

• your-plugin will run and return children
• some-other-plugin will run only on those child nodes and the current node will be treated as a grouping node (no inputs)

Does that make sense?

[jcendava]

Yes. Thank you for the clarification.