Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to transform & split results once recieved #8

Open
jhorbulyk opened this issue Dec 3, 2019 · 0 comments
Open

Add the ability to transform & split results once recieved #8

jhorbulyk opened this issue Dec 3, 2019 · 0 comments

Comments

@jhorbulyk
Copy link

Implements the functionality described in https://github.com/elasticio/elasticio/issues/2793 at a component level (assuming that it is not implemented at a platform level).

SOAP equivalent of this functionality in the REST Component: elasticio/rest-api-component#91 & elasticio/rest-api-component#68

Consider the following flow case:
Simple Trigger -> SOAP API call that returns large array -> Splitter -> ...

Imagine that the SOAP API call returns an array with total size 1 MB with 9000 entries. Given that for each object produced by Splitter includes the entire result of the SOAP API call in every message emitted from the splitter, the above flow would produce 9 GB of data on execution easily exceeding the queue size limits. (This is further exasterbasted by elasticio/sailor-nodejs#80)

However, if these messages are 100% independent of each other, then there is no reason to attach each result to the larger SOAP body. It would be complex to change passthrough logic or message structure to handle the above case. In theory, the above could be tuned to emit things slower, but that also introduces problems. However, if we understand that the above is a common pairing, then it may make sense to add the following two inputs to the SOAP API component:

  • Body JSONata transform expression (optional, default to idempotent $)
  • Split result if it is an array (optional, default to false)

The component behavior would be changed to:
After the component has a JSON representation of the response body that they are going to emit, then:

  1. Evaluate the provided JSON expression if it exists to the body of the HTTP response.
  2. If the result is an array and splitting is true, emit a message per array entry. Otherwise emit the transformed result.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant