feat!: allow retrieving all billing accounts and add billing_information column to gcp_project and gcp_organization_project tables

[pdecat]

This is a breaking change, submitted as draft to initiate a discussion.

The rationale of this PR is that a billing account is not tied to a single project.
Without this change, all existing billing accounts cannot be discovered.

Edit: rebased on main so removed comment about what commits to review.

Integration test logs

Logs

Add passing integration test logs here

Example query results

Results

> select name, master_billing_account from gcp_billing_account order by master_billing_account, name
+----------------------+--------------------------------------+
| name                 | master_billing_account               |
+----------------------+--------------------------------------+
| 00D010-012345-000000 |                                      |
| 011C92-012345-111111 |                                      |
| 016C27-012345-222222 |                                      |
| 01BDFD-012345-333333 |                                      |
| 01C328-012345-444444 |                                      |
| 0024B1-012345-012345 | billingAccounts/00D010-012345-000000 |
| 010892-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0109E7-012345-012345 | billingAccounts/00D010-012345-000000 |
| 010FA8-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0110A7-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0112CD-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01245D-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0128B9-012345-012345 | billingAccounts/00D010-012345-000000 |
| 012DD3-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0130FA-012345-012345 | billingAccounts/00D010-012345-000000 |
| 013522-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0145D3-012345-012345 | billingAccounts/00D010-012345-000000 |
| 014F17-012345-012345 | billingAccounts/00D010-012345-000000 |
| 015116-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0151FE-012345-012345 | billingAccounts/00D010-012345-000000 |
| 015574-012345-012345 | billingAccounts/00D010-012345-000000 |
| 016C91-012345-012345 | billingAccounts/00D010-012345-000000 |
| 017047-012345-012345 | billingAccounts/00D010-012345-000000 |
| 0180B0-012345-012345 | billingAccounts/00D010-012345-000000 |
| 018C5C-012345-012345 | billingAccounts/00D010-012345-000000 |
| 018D0A-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01A1D3-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01BAE2-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01C9B8-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01C9CA-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01DE4B-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01E25A-012345-012345 | billingAccounts/00D010-012345-000000 |
| 01E62F-012345-012345 | billingAccounts/00D010-012345-000000 |
| 014D64-012345-012345 | billingAccounts/01F621-012345-555555 |
| 016524-012345-012345 | billingAccounts/01F621-012345-555555 |
| 01DE7B-012345-012345 | billingAccounts/01F621-012345-555555 |
| 01FD31-012345-012345 | billingAccounts/01F621-012345-555555 |
+----------------------+--------------------------------------+

[misraved]

Hey @pdecat, could you please resolve the merge conflicts in this PR?

Also, is the PR waiting on some discussion? Apologies if I missed any thread.

[pdecat]

Rebased on main and marked as ready.

Also, is the PR waiting on some discussion? Apologies if I missed any thread.

Not from my point of view, I just wanted to emphasize that this is somewhat a breaking change.

[cbruno10]

@pdecat Can you please describe a bit more in detail what the breaking changes are? What types of queries would be affected?

[pdecat]

This PR changes the table behavior by returning all billing accounts accessible by the connection's credentials (multiple rows).
Previously, only the billing account attached to the project that is configured for the connection was returned (always at most one row).

[cbruno10]

@pdecat So if I understand correctly, this PR:

• Updates the gcp_billing_account table so it returns all billing accounts the connection/credentials has access to, instead of just the current project in the connection (breaking change).
• Removes the project column from the gcp_billing_account table (breaking change).
• Adds the billing_information column to the gcp_organization_project and gcp_project tables (new feature).

Some questions:

• Does the https://cloud.google.com/billing/docs/reference/rest/v1/organizations.billingAccounts/list API call return a subset (limited by organization) of https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list (which is the API call this PR is using I believe)? It seems like using https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list is the correct list API call, but just double checking.
• Are billing subaccounts returned by the gcp_billing_account table by default? The docs say you can set these in the filter(https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list) or there's a separate API call for them (https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts.subAccounts/list), but if they're just billing accounts in shape, then I'm not sure we'd need to use the separate API call.

Also, for a future addition, I think we could add a gcp_billing_account_project table that uses https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts.projects/list, which would allow users to view all projects associated with a particular billing account. This would be a separate table as opposed to a projects column in the gcp_billing_account_project table since it has paging. But, I'm not suggesting we add that as part of this PR and we can wait until someone has a particular use case.

[pdecat]

@pdecat So if I understand correctly, this PR:

• Updates the gcp_billing_account table so it returns all billing accounts the connection/credentials has access to, instead of just the current project in the connection (breaking change).
• Removes the project column from the gcp_billing_account table (breaking change).
• Adds the billing_information column to the gcp_organization_project and gcp_project tables (new feature).

Exactly. Forgot about the second breaking change in my previous message.

Some questions:

• Does the https://cloud.google.com/billing/docs/reference/rest/v1/organizations.billingAccounts/list API call return a subset (limited by organization) of https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list (which is the API call this PR is using I believe)? It seems like using https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list is the correct list API call, but just double checking.

Good question. The only apparent difference I could find between the two endpoints is that the optional parent parameter is passed in the path in the former instead of a query parameter in the latter.

• Are billing subaccounts returned by the gcp_billing_account table by default? The docs say you can set these in the filter(https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts/list) or there's a separate API call for them (https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts.subAccounts/list), but if they're just billing accounts in shape, then I'm not sure we'd need to use the separate API call.

With this PR, yes. I've just added an example query with results in this PR's description. No need for a separate API call I believe.

Also, for a future addition, I think we could add a gcp_billing_account_project table that uses https://cloud.google.com/billing/docs/reference/rest/v1/billingAccounts.projects/list, which would allow users to view all projects associated with a particular billing account. This would be a separate table as opposed to a projects column in the gcp_billing_account_project table since it has paging. But, I'm not suggesting we add that as part of this PR and we can wait until someone has a particular use case.

👍

[github-actions]

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[pdecat]

Not stale.

[cbruno10]

@pdecat Sorry for dropping off on this PR. @misraved and I were discussing the PR this morning and as part of our deprecation and removal strategies, if it'd be better to deprecate the project column from the gcp_billing_account table first, and then in a future date, remove it (as part of v2.0.0). Typically, we deprecate tables or columns set to be removed to give users a chance to update their queries or mods (especially in Pipes).

The information in that column would still just return the active project (which is probably the one specified in the connection or set in the CLI), so it's not very useful, but this would be consistent with the project columns in other tables and users could still get the current billing account by the name/display_name columns.

Thoughts?

[pdecat]

Hi @cbruno10, I've re-added the project column and appended (Deprecated) to its description.

[cbruno10]

Thanks again for the changes @pdecat !