BigQueryIO uniformize direct and export reads #32360

RustedBones · 2024-08-29T13:13:53Z

Refers to #26329, also fix #20100, #21076

When using readWithDatumReader and DIRECT_READ method, the transform would fail because the parseFn is expected. Refactor the IO so the avro datumReader can be use in both cases.

In some case, it is required to get the data with the desired schema. Currently, BQ io always uses the writer schema (or table schema). Create new APIs to set the reader schema.

~~This refactoring contains some breaking changes:~~

withFormat is not exposed anymore. Indeed, it is not possible to configure a TypedRead with a DatumReaderFactory and change the format later. Data format MUST be chosen when creating the transform.

In the TypedRead.Builder, replace the DatumReaderFactory with the BigQueryReaderFactory allowing to handle both avro and arrow in uniform fashion. This alters the BigQueryIOTranslation.
I need some help on that point to handle that in a better way.

Edit: reworked part of this PR to keep compatibility

github-actions · 2024-08-29T14:36:02Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

RustedBones · 2024-08-30T11:37:50Z

assign set of reviewers

github-actions · 2024-08-30T11:38:59Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @robertwb for label java.
R: @ahmedabu98 for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

RustedBones · 2024-08-30T11:40:49Z

Some BQ integration tests are failing.

I don't know schema & data of the following big_query_import_export.parallel_read_table_row_xxx tables so I can recreated the setup in a personal GCP project. Can someone give me a hand ?

RustedBones · 2024-09-04T08:14:32Z

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

+      // read table schema and infer coder if possible
+      Coder<T> c;
+      if (getCoder() == null) {
+        tableSchema = requestTableSchema(sourceDef, bqOptions, getSelectedFields());


Is it fine to access the BQ table at graph creation time? (It was already doing that when beam schema was requested)

Yeah this is a valid concern. I've heard use case where pipeline submission machine does not or has incomplete permission to the resource, and infer schema at graph creation time can cause issue. General guideline is the use case used to work should be able to work still (and vice versa)

RustedBones · 2024-09-04T08:16:57Z

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

@@ -1731,7 +1870,7 @@ public void processElement(ProcessContext c) throws Exception {
                                          .setTable(
                                              BigQueryHelpers.toTableResourceName(
                                                  queryResultTable.getTableReference()))
-                                          .setDataFormat(DataFormat.AVRO))


was arrow even supported ?

...-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryReaderFactory.java

...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceDef.java

...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java

...-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTranslation.java

RustedBones · 2024-09-04T08:44:37Z

...-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTranslation.java

@@ -89,8 +95,8 @@ static class BigQueryIOReadTranslator implements TransformPayloadTranslator<Type
            .addNullableBooleanField("use_legacy_sql")
            .addNullableBooleanField("with_template_compatibility")
            .addNullableByteArrayField("bigquery_services")
-            .addNullableByteArrayField("parse_fn")
-            .addNullableByteArrayField("datum_reader_factory")
+            .addNullableByteArrayField("bigquery_reader_factory")


This is a complex object to serialize. subject to serialization error if there's changes between versions

...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java

github-actions · 2024-09-11T12:14:19Z

Reminder, please take a look at this pr: @robertwb @ahmedabu98

Reduce breaking changes by configuring IO with simple objects

Abacn · 2025-01-09T18:32:32Z

waiting on author

Abacn · 2025-01-09T18:33:45Z

Looks like this is a evolving PR, please let me know when it's ready. Also, good to split independent fix out so it expedites review. Thank you!

RustedBones · 2025-01-10T08:07:08Z

This is not evolving anymore.
I rebased many times the PR to re-trigger the CI, in the hope to get all green checks in CI, without success.
Seems most of the time the error is unrelated to the changes.

Will merge master in the future. It will be visible on your end that no changes are added.

github-actions · 2025-01-17T12:14:08Z

Reminder, please take a look at this pr: @robertwb @Abacn

github-actions · 2025-01-21T12:14:40Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @kennknowles for label java.
R: @ahmedabu98 for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

github-actions · 2025-01-28T12:14:46Z

Reminder, please take a look at this pr: @kennknowles @ahmedabu98

github-actions · 2025-01-31T12:14:32Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @damondouglas for label java.
R: @Abacn for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

github-actions · 2025-02-08T12:14:15Z

Reminder, please take a look at this pr: @damondouglas @Abacn

github-actions · 2025-02-12T12:14:52Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @robertwb for label java.
R: @johnjcasey for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

Abacn · 2025-02-12T15:27:18Z

sorry for being late and thanks again for persistence. If there are something most urgent (and small) within this PR, feel free to send smaller PRs for review. Had been challenging to find sufficient chunk of time reviewing large changes.

github-actions bot added java io extensions gcp zetasketch labels Aug 29, 2024

RustedBones force-pushed the bq-read-schema branch from 5dbe389 to 989ccdd Compare August 30, 2024 08:59

github-actions bot added the examples label Aug 30, 2024

github-actions bot added the Next Action: Reviewers label Aug 30, 2024

github-actions bot added examples and removed examples labels Aug 30, 2024

RustedBones marked this pull request as draft September 3, 2024 07:18

github-actions bot added examples and removed examples labels Sep 3, 2024

RustedBones marked this pull request as ready for review September 3, 2024 19:48

github-actions bot added examples and removed examples labels Sep 4, 2024

RustedBones commented Sep 4, 2024

View reviewed changes

...-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTranslation.java Outdated Show resolved Hide resolved

RustedBones commented Sep 4, 2024

View reviewed changes

...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java Outdated Show resolved Hide resolved

github-actions bot added the slow-review label Sep 11, 2024

RustedBones added 4 commits January 9, 2025 12:29

Remove avro parameter from generic interface

783111f

Reduce breaking changes

987da27

Reduce breaking changes by configuring IO with simple objects

Handle non serializable avro Schema

68daadf

Consistent naming with AvroIO

68d0c72

RustedBones force-pushed the bq-read-schema branch from 1c52b2a to 68d0c72 Compare January 9, 2025 11:29

github-actions bot added kotlin and removed kotlin labels Jan 9, 2025

github-actions bot added Next Action: Author and removed Next Action: Reviewers labels Jan 9, 2025

github-actions bot added Next Action: Reviewers and removed Next Action: Author labels Jan 10, 2025

Merge remote-tracking branch 'upstream/master' into bq-read-schema

496945b

github-actions bot added kotlin and removed kotlin labels Jan 10, 2025

github-actions bot added the slow-review label Jan 17, 2025

github-actions bot removed the slow-review label Jan 21, 2025

github-actions bot added the slow-review label Jan 28, 2025

github-actions bot removed the slow-review label Jan 31, 2025

github-actions bot added the slow-review label Feb 8, 2025

github-actions bot removed the slow-review label Feb 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BigQueryIO uniformize direct and export reads #32360

BigQueryIO uniformize direct and export reads #32360

RustedBones commented Aug 29, 2024 •

edited

Loading

github-actions bot commented Aug 29, 2024

RustedBones commented Aug 30, 2024

github-actions bot commented Aug 30, 2024

RustedBones commented Aug 30, 2024

RustedBones Sep 4, 2024

Abacn Sep 16, 2024

RustedBones Sep 4, 2024

RustedBones Sep 4, 2024

github-actions bot commented Sep 11, 2024

Abacn commented Jan 9, 2025

Abacn commented Jan 9, 2025

RustedBones commented Jan 10, 2025 •

edited

Loading

github-actions bot commented Jan 17, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 28, 2025

github-actions bot commented Jan 31, 2025

github-actions bot commented Feb 8, 2025

github-actions bot commented Feb 12, 2025

Abacn commented Feb 12, 2025

BigQueryIO uniformize direct and export reads #32360

Are you sure you want to change the base?

BigQueryIO uniformize direct and export reads #32360

Conversation

RustedBones commented Aug 29, 2024 • edited Loading

github-actions bot commented Aug 29, 2024

RustedBones commented Aug 30, 2024

github-actions bot commented Aug 30, 2024

RustedBones commented Aug 30, 2024

RustedBones Sep 4, 2024

Choose a reason for hiding this comment

Abacn Sep 16, 2024

Choose a reason for hiding this comment

RustedBones Sep 4, 2024

Choose a reason for hiding this comment

RustedBones Sep 4, 2024

Choose a reason for hiding this comment

github-actions bot commented Sep 11, 2024

Abacn commented Jan 9, 2025

Abacn commented Jan 9, 2025

RustedBones commented Jan 10, 2025 • edited Loading

github-actions bot commented Jan 17, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 28, 2025

github-actions bot commented Jan 31, 2025

github-actions bot commented Feb 8, 2025

github-actions bot commented Feb 12, 2025

Abacn commented Feb 12, 2025

RustedBones commented Aug 29, 2024 •

edited

Loading

RustedBones commented Jan 10, 2025 •

edited

Loading