ntd_id: revise dim organiztions and enrich ntd endpoints

[charlie-costanzo]

Description

This PR seeks to replace the deprecated use of _deprecated__ntd_agency_to_organization in dim_organizations with the appropriate alternative, as well as enrich ntd mart tables with caltrans_district from the refactored dim_organizations, and perform some other cleanup tasks as outlined below:

Refactor mart.dim_organizations:

• remove deprecated use of _deprecated__ntd_agency_to_organization
• instead, utilize ntd_agency_info_key from int_transit_database__organizations_dim for ntd_id previously inherited from _deprecated__ntd_agency_to_organization
• Remaining Questions
  • Is this an appropriate substitution and use of ntd_id?
  • Should we be incorporating versioned use of ntd_id/ntd_agency_info_key based on analysis period? Thinking specifically in relation to cutover date and our dashboarding work.
  • Or is analysis best performed with current ntd_id and caltrans_district

Enrich NTD mart tables with caltrans_district:

• In the following directories: mart.ntd_annual_reporting; mart.ntd_funding_and_expenses; mart.ntd_ridership; mart.ntd_safety_and_security;

Data type handling in stg_transit_database__organizations:

• Cast unnested_ntd_records/ntd_agency_info_key as string appropriately

Rename column in stg_ntd__major_safety_events / fct_major_safety_events:

• rename ntdid as ntd_id for consistency

Fix incorrectly named CTEs, add explicit column names (dbt best practice):

• mart.ntd_ridership
  • TODO: consolidate ntd ridership directories currently duplicated and split between ntd and ntd_ridership directories
  • This should probably mean continuing to use the tables currently in the ntd directory but moving them to the appropriately-specific directory ntd_rideship

Resolves: #3709

Type of change

• Bug fix (non-breaking change which fixes an issue)

How has this been tested?

poetry run dbt run -s +mart.ntd_ridership +mart.ntd_annual_reporting +mart.ntd_safety_and_security +mart.funding_and_expenses +mart.transit_database

Post-merge follow-ups

• No action required

[evansiroky]

I believe this is still extracting the Caltrans district from dim_organizations which extracts it form a deprecated column in Airtable. We should probably remove Caltrans District from the dim_organizations table.

[charlie-costanzo]

Ah okay, thanks for letting me know @evansiroky! I was hoping to use this to enrich the new NTD mart tables with caltrans_district on ntd_id, is there a different source of truth for caltrans_district that we could use?

[evansiroky]

See this Caltrans District enrichment of Dim Organizations Demo metabase question