Handle target rows case (Filter, Switch/Case, ...)

[mattcasters]

No description provided.

[mattcasters]

Added test-cases for Filter Rows and Switch/Case, the most popular ones.
I have now confidence the generics work now behind this.

[twknight]

This appears to be broken for me (using latest 0.6/Master versions). With the Filter Rows step, I can only get output using the Main Output from the step. If I direct any rows (True or False), no rows are output (even with one still using Main Output). See the following:

It seems like an issue with creating the additional PCollection outputs, as the graph shows the same tag ("out0") for both targets' input step. I'm not sure why creating one target (a False result in my example) destroys the records for the step's Main Output, but that's what I'm observing.

[mattcasters]

To be fair, that filter step is not configured correctly (otherwise said, something I have never seen, didn't know it was working :-))

So either just filter the rows with a default output or user a True/False split.

[twknight]

Sorry, I didn't include the most obvious case in my example, but a True/False split isn't working either, which is what led me to try the Main Output. I've completely removed the step, saved, tried again...same results each time.

[mattcasters]

Should be easy to fix.

[mattcasters]

Is filter-split-combine.ktr working for you?

[mattcasters]

Also, this is an old case, it got me confused for a while.
If you have bugs, please simply report them, the more the better, TIA :-)

[mattcasters]

Well, FWIW, both the split and the filter-default.ktr example (I just added it to kettle-beam-examples) works fine. Try to update to kettle-beam v1.0.0 and recompile it.

[twknight]

filter-split-combine.ktr is working correctly. I also modified it to filter on the integer id column, which also worked. Only obvious difference I'm seeing is the BigQuery source in mine, but hard to imagine that Kettle sees the records any differently when they arrive to the step.

I'll get v1.0.0 and try again. Thanks for your help.

[twknight]

It appears the TRUE/FALSE targeted output only works if going to another step prior to Beam Output. Simply adding a Dummy step between the Filter and the outputs allowed it to work. Perhaps the grouping/flattening taking place in Beam Output is hard-coded to the default PCollection?

[mattcasters]

Sorry, I can't confirm this. Works just fine on my end.
I created test-transformation "filter-direct-output.ktr" in the kettle-beam-examples project

[virivigio]

Actually it still happens on kettle-neo4j-remix-beam-8.2.0.7-719-REMIX.zip
"filter-direct-output.ktr" is affected as well, just remove the "Add constants" steps

[mattcasters]

Urgh, you're right. It regressed somewhere along the line. The pipeline is created correctly but the data passing isn't happening.

To catch it you would have to look at the metrics.