[COST-5137] Azure managed summary

[myersCody]

Jira Ticket

COST-5137

Description

This change will:

1. Break out our complex OCP on CLOUD daily summary logic into separate files.

Instead of having one long 1500 line sql file, we have different stages of progress.

1. Create table
2. Resource Matching
3. Daily summary <-- daily aggregation

During the daily aggregation flow we also handle some data transformation such as:

1. Unattributed storage
2. Unattributed network

As the features we support continue to grow, the size of these files will as well. Therefore, we now have the ability to chunk out the files as we see fit. Personally in the future I would like to break out the network & storage unattributed data transformations from the daily aggregation logic. But that is outside the scope of this issue.

Testing

1. Checkout Branch
2. Load test customer data
3. Compare tables:

select sum(pretax_cost), resource_id, array_agg(distinct matched_tag) from managed_reporting_ocpazurecostlineitem_project_daily_summary where year='2024' and month='12' group by resource_id order by resource_id;

select sum(pretax_cost), resource_id from reporting_ocpazurecostlineitem_project_daily_summary where year='2024' and month='12' group by resource_id order by **resource_id;

Example:

trino:org1234567> select sum(pretax_cost), resource_id, array_agg(distinct matched_tag) from managed_reporting_ocpazurecostlineitem_project_daily_summary where year='2024' and month='12' group by resource_id order by resource_id;
       _col0        |               resource_id               |                                     _col2
--------------------+-----------------------------------------+-------------------------------------------------------------------------------
  4.002846943148718 | azure-cloud-prefix-pvc-partial-matching | [NULL]
        9.324266345 | azure_compute1                          | [NULL]
  4.219567560000001 | azure_compute1_OSDisk                   | [NULL]
        4.595152264 | azure_compute2                          | [NULL]
  4.228297566999999 | azure_compute3                          | [NULL]
        5.561305184 | azure_master                            | [NULL]
  4.855337400895995 | disk-id-1234567                         | [NULL]
 3.2741646441796077 | pv-123-claimless                        | [NULL]
        28.87511884 | NULL                                    | ["app": "banking", "app": "weather", "storageclass": "Thor", "app": "mobile"]
(9 rows)

trino:org1234567> select sum(pretax_cost), resource_id from reporting_ocpazurecostlineitem_project_daily_summary where year='2024' and month='12' group by resource_id order by resource_id;
       _col0        |               resource_id
--------------------+-----------------------------------------
  4.002846943148717 | azure-cloud-prefix-pvc-partial-matching
  9.324266344999998 | azure_compute1
         4.21956756 | azure_compute1_OSDisk
  4.595152264000001 | azure_compute2
  4.228297566999999 | azure_compute3
  5.561305184000001 | azure_master
  4.861264507315131 | disk-id-1234567
 3.2741646441796073 | pv-123-claimless

You will notice an additional cost that doesn't have a resource_id. That cost is related to tag matching rows. Apparently tag matching just doesn't work in our old flow.

Release Notes

• proposed release note

* [COST-####](https://issues.redhat.com/browse/COST-####) Fix some things

[myersCody]

/retest

[codecov]

Codecov Report

Attention: Patch coverage is 94.33962% with 3 lines in your changes missing coverage. Please review.

Project coverage is 94.1%. Comparing base (fb3f92f) to head (d19d694).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #5475     +/-   ##
=======================================
- Coverage   94.1%   94.1%   -0.0%     
=======================================
  Files        371     371             
  Lines      31624   31638     +14     
  Branches    3387    3388      +1     
=======================================
+ Hits       29756   29766     +10     
- Misses      1217    1218      +1     
- Partials     651     654      +3     

[myersCody]

https://issues.redhat.com/browse/COST-5866

[myersCody]

If we want to rename these tables to better match our flow now is our chance. For example this table is where we start doing resource matching.

[lcouzens]

Naming is always fun. Do you have suggestions here at all?

[myersCody]

I guess what I have named them so far 😂 Mostly wanted to call it out in case people had preferences.