Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery cloud test for Iceberg catalogs #24969

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

Conversation

rpdevmp
Copy link
Contributor

@rpdevmp rpdevmp commented Jan 29, 2025

Summary:

  • Added BigQuery cloud test for Iceberg catalogs
  • Currently it is using "key_value" Iceberg mode
  • It supports using AWS S3 storage or GS GCP storage
  • Tested multiple times with Ducktape on GCP cloud clusters
  • Also, tested with bq tool on GCP and AWS clusters
  • Running simple queries to verify Iceberg catalog data

Backports Required

  • [x ] none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.3.x
  • v24.2.x
  • v24.1.x

Release Notes

  • none

@rpdevmp rpdevmp force-pushed the rpdevmp/DEVPROD-2430 branch from fb5ab7c to 9906648 Compare January 29, 2025 06:08
@rpdevmp
Copy link
Contributor Author

rpdevmp commented Jan 29, 2025

  • Ducktape tests were done on GCP BYOC cluster, with Iceberg enabled, key_value iceberg mode
  • Need to add google-cloud-bigquery to Ducktape requirements, so its available on Ducktape node. As a workaround, used this command before running tests:

sudo /opt/.ducktape-venv/bin/python3 -m pip install google-cloud-bigquery

But this is NOT a blocker for this PR. We should be able to review and merge it as tests work

As a follow up, we could extend it with other supported modes, more topics, different data formats, etc. But those enhancements are NOT part of this PR.

@rpdevmp rpdevmp marked this pull request as ready for review January 29, 2025 06:22
@rpdevmp rpdevmp requested review from andrwng and andrewhsu January 29, 2025 06:22
@rpdevmp
Copy link
Contributor Author

rpdevmp commented Jan 29, 2025

@rpdevmp rpdevmp changed the title Rpdevmp/devprod 2430 Added BigQuery cloud test for Iceberg catalogs Jan 29, 2025
@rpdevmp rpdevmp changed the title Added BigQuery cloud test for Iceberg catalogs BigQuery cloud test for Iceberg catalogs Jan 29, 2025
@rpdevmp rpdevmp force-pushed the rpdevmp/DEVPROD-2430 branch from 9906648 to b022ea6 Compare January 29, 2025 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant