Filter Data in stage macro

[MatthiasWendtB24]

Hi Folks,
is there a simple solution, to filter the rows processed by the stage macro?

Thanks for your response,
Kind regards,
Matthias

[tkirschke]

Hi @MatthiasWendtB24 and thanks for reaching out!

On which condition would you like to filter the rows?
Without any more details, I would suggest creating a dbt model that queries your source and applies the filter condition. Then I would recommend configuring this model to be materialized as "ephemeral".
Within your stage model, you then define this ephemeral model as the source model of your stage. Now only the filtered rows should be processed by the stage macro.

Let me know if this helps!
Best regards,
Tim

[MatthiasWendtB24]

Hi @tkirschke ,
thank you for your response. That what you descripe is currently my workaround, i build a model where i filter my stage table with a single where-Condition, it would be super handy to have an macro-parameter, where i apply the filter condition. That would save me an extra (ephemeral) model.

Best regards,
Matthias

[MatthiasWendtB24]

Hi @tkirschke ,
i worked a little bit around your suggestion with the ephemeral model and i think, that doesn't work, because such models couldn't be use in macro calls. So i fall back to my first solution to build filtered views

[bschlottfeldt]

Might be a good idea for enhancement: allow an optional parameter in the yaml metadata of the stage called filter_source_by and allow the user to add a statement as the input.
We can add this to the list of features to be developed for enhancement, but I can't give a timeline on when it will be done as we have other features in the pipeline.

Kind regards,
Barbara