Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix streaming support for multi property configurations #257

Merged
merged 5 commits into from
Aug 2, 2023
Merged

Fix streaming support for multi property configurations #257

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c20d5b1
remove check that enables intraday support for only single-property p…
adamribaudo-velir Jul 30, 2023
f062f06
Revert "remove check that enables intraday support for only single-pr…
adamribaudo-velir Jul 30, 2023
6a428ae
include intraday data in the source select statement
adamribaudo-velir Jul 30, 2023
ff905cc
remove unnecessary source
adamribaudo-velir Jul 30, 2023
ae16202
fix incremental condition
adamribaudo-velir Jul 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 9 additions & 35 deletions models/staging/base/base_ga4__events.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,7 @@
{% for i in range(var('static_incremental_days')) %}
{% set partitions_to_replace = partitions_to_replace.append('date_sub(current_date, interval ' + (i+1)|string + ' day)') %}
{% endfor %}
{% if var('property_ids', false) == false %}
{% set relations_intraday = dbt_utils.get_relations_by_pattern(schema_pattern=var('dataset'), table_pattern='events_intraday_%', database=var('project')) %}
{% endif %}

{{
config(
pre_hook="{{ ga4.combine_property_data() }}" if var('property_ids', false) else "",
Expand All @@ -19,44 +17,20 @@
)
}}

with source_daily as (
with source as (
select
{{ ga4.base_select_source() }}
from {{ source('ga4', 'events') }}
where _table_suffix not like '%intraday%'
and cast( _table_suffix as int64) >= {{var('start_date')}}
where cast( replace(_table_suffix, 'intraday_', '') as int64) >= {{var('start_date')}}
{% if is_incremental() %}
and parse_date('%Y%m%d', left(_TABLE_SUFFIX, 8)) in ({{ partitions_to_replace | join(',') }})
and parse_date('%Y%m%d', left( replace(_table_suffix, 'intraday_', ''), 8)) in ({{ partitions_to_replace | join(',') }})
{% endif %}
),
-- Include intraday data if using a single-property configuration and the events_intraday_* table exists
{% if var('property_ids', false) == false and relations_intraday|length > 0 %}
source_intraday as (
select
{{ ga4.base_select_source() }}
from {{ source('ga4', 'events_intraday') }}
where cast( _table_suffix as int64) >= {{var('start_date')}}
{% if is_incremental() %}
and parse_date('%Y%m%d', left(_TABLE_SUFFIX, 8)) in ({{ partitions_to_replace | join(',') }})
{% endif %}
),
unioned as (
select * from source_daily
union all
select * from source_intraday
),
renamed as (
select
{{ ga4.base_select_renamed() }}
from unioned
)
{% else %}
renamed as (
select
{{ ga4.base_select_renamed() }}
from source_daily
)
{% endif%}
renamed as (
select
{{ ga4.base_select_renamed() }}
from source
)

select * from renamed
qualify row_number() over(partition by event_date_dt, stream_id, user_pseudo_id, session_id, event_name, event_timestamp, to_json_string(event_params)) = 1
5 changes: 1 addition & 4 deletions models/staging/src_ga4.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,4 @@ sources:
tables:
- name: events
identifier: events_* # Scan across all sharded event tables. Use the 'start_date' variable to limit this scan
description: Main events table exported by GA4. Sharded by date.
- name: events_intraday
identifier: events_intraday_*
description: Intraday events table which is optionally exported by GA4.
description: Main events table exported by GA4. Sharded by date.
Loading