Releases: spotify/scio
Releases · spotify/scio
v0.14.10
Includes Beam 2.61.0 support.
This release also comes with a better support of BQ types. New TableRow extension methods were added for those types.
To note the behavior change of (r: TableRow).getRecord("field-name") that now throws instead of returning null when field-name is not present in the row. Change for getRecordOpt(f: AnyRef): Option[TableRow] instead.
🚀 Enhancements
- Support Snowflake (#5500) by @turb (#5502)
- Refactor partitionByKey to get clearer error (#5527) @RustedBones
- Allow timeout for async DoFn (#5534) @RustedBones
- Update default Dataflow add-open opts for Java 17+ (#5537) @clairemcginty
- Include more default jdkAddOpenModules for Java 17+ (#5538) @clairemcginty
- Add Batched lookups for streaming GRPC endpoints and BigTable (#5521) @lofifnc
🐛 Bug Fixes
- BigQuery JSON column: encode as Jackson JsonNode (#5523) @turb
- Stable TableRow converted from BQ types (#5536) @RustedBones
📗 Documentation
- Prepare for Scio 0.14.10 (#5539) @clairemcginty
🏗️ Build Improvements
- Skip BQ avro typed integration test until 2.61 (#5526) @RustedBones
🧪 Test Improvements
- best effort cleanup (#5528) @RustedBones
🌱 Dependency Updates
- Update to BigQuery read API v1 interface (#5431) @kmjung
- Update beam to 2.61 (#5525) @RustedBones
- Bump codecov/codecov-action from 4 to 5 (#5524) @dependabot
- Update sbt-scalafix to 0.13.0 (#5506) @scala-steward
- Update tensorflow-metadata to 0.16.1 (#5535) @RustedBones
Contributors to this release
@RustedBones, @turb, @kmjung, @lofifnc, @clairemcginty and @scala-steward
New Contributors
Full Changelog: v0.14.9...v0.14.10
Contributors
turb, clairemcginty, and 5 other contributors
Assets 3
v0.14.9
What's changed
Includes Beam 2.60.0 support.
🚀 Enhancements
- Add missing fanout SCollection API (#5497) @RustedBones
- Add latest API to SCollection (#5495) @RustedBones
- Add top to collections with fanout (#5487) @RustedBones
🐛 Bug Fixes
- Fix RemoteFileUtil to download in parallel as expected. (#5515) @psobot
- Fix file batch operation (#5519) @RustedBones
📗 Documentation
- Prepare for release 0.14.9 (#5522) @RustedBones
🏗️ Build Improvements
- Remove deprecated usage of tlSonatypeUseLegacyHost in build (#5517) @RustedBones
- Update sbt-bom to 1.0.16 (#5501) @scala-steward
- Update sbt-mdoc to 2.6.1 (#5496) @scala-steward
- Update sbt, sbt-dependency-tree to 1.10.2 (#5490) @scala-steward
🔧 Refactorings
- Consistent implementation for reduce operation (#5498) @RustedBones
🌱 Dependency Updates
- Update beam to 2.60 (#5516) @RustedBones
- Update socco-ng to 0.1.13 (#5518) @RustedBones
- Update scala-compiler, scala-library, ... to 2.13.15 (#5504) @scala-steward
- Update elasticsearch-java to 8.15.2 (#5503) @scala-steward
- Update jedis to 5.2.0 (#5507) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.21.0 (#5509) @scala-steward
- Update metrics-core to 4.2.28 (#5511) @scala-steward
- Update scalacheck to 1.18.1 (#5491) @scala-steward
- Update jna to 5.15.0 (#5489) @scala-steward
- Update munit to 1.0.2 (#5492) @scala-steward
Contributors to this release
Contributors
psobot, RustedBones, and scala-steward
Assets 3
v0.14.8
What's changed
Includes Beam 2.59.0 support.
🐛 Bug Fixes
- (fix #5472) Set desiredBundleSizeBytes to Long.MaxValue BinaryIO reads to prevent file subrange-splitting (#5473) @clairemcginty
- use noOp cache instead of null when instantiating typedSparkeySideInput (#5469) @pgoggijr
📗 Documentation
- Prepare release 0.14.8 (#5486) @RustedBones
- Fix redis examples (#5482) @kellen
- Fixes a minor formatting issue in the FAQs (#5467) @luciomartinez
🧪 Test Improvements
- Add test for empty string arg (#5485) @RustedBones
- Support pubsub epoch timestamp attribute in test (#5484) @RustedBones
🏗️ Build Improvements
- Update sbt-typelevel to 0.7.3 (#5480) @scala-steward
- Update sbt-scoverage to 2.2.0 (#5479) @scala-steward
- Update sbt-avro to 3.5.0 (#5461) @scala-steward
🌱 Dependency Updates
- Update beam to 2.59 (#5471) @RustedBones
- Update elasticsearch-java to 8.15.1 (#5474) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.20.1 (#5475) @scala-steward
- Update circe-core, circe-generic, ... to 0.14.10 (#5476) @scala-steward
- Update scala-compiler, scala-library, ... to 2.12.20 (#5478) @scala-steward
- Update parquet-avro, parquet-column, ... to 1.14.2 (#5477) @scala-steward
- Update metrics-core to 4.2.27 (#5463) @scala-steward
- Update mssql-jdbc to 12.8.1.jre11 (#5462) @scala-steward
- Update jedis to 5.1.5 (#5465) @scala-steward
Contributors to this release
@RustedBones, @clairemcginty, @kellen, @luciomartinez, @pgoggijr and @scala-steward
Contributors
kellen, clairemcginty, and 4 other contributors
Assets 3
v0.14.7
What's changed
Includes Beam 2.58.1 support.
🚀 Enhancements
🐛 Bug Fixes
- Fix Async DoFn to emit element with restored metadata (#5441) @RustedBones
📜 Scalafix Migrations
- Improve accuracy of FixQuery rule (#5438) @clairemcginty
📗 Documentation
- Update version matrix with scio 0.14.7 (#5460) @RustedBones
🏗️ Build Improvements
- Ignore updates from zstd-jni and sync with beam's version (#5457) @RustedBones
- Update scalac-options to 0.1.7 (#5454) @scala-steward
- Update scalafmt-core to 3.8.3 (#5453) @scala-steward
- Move files in folder matching package (#5445) @RustedBones
- Enforce javafmt on CI (#5442) @RustedBones
- Disable MiMa for scio-bom (#5439) @RustedBones
🌱 Dependency Updates
- Update beam to 2.58.1 (#5459) @RustedBones
- Update elasticsearch-java to 8.15.0 (#5446) @scala-steward
- Update jedis to 5.1.4 (#5455) @scala-steward
- Update munit to 1.0.1 (#5452) @scala-steward
- Update voyager to 2.0.9 (#5450) @scala-steward
- Update mssql-jdbc to 12.8.0.jre11 (#5449) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.20.0 (#5448) @scala-steward
- Update beam to 2.58 (#5436) @RustedBones
Contributors to this release
Contributors
clairemcginty, RustedBones, and 2 other contributors
Assets 3
v0.14.6
What's changed
Includes Beam 2.57.0 support.
🚀 Enhancements
- Publish scio-bom (#5423) @RustedBones
- Support BigQuery JSON column type (#5416) @turb
🐛 Bug Fixes
- fix: use fully qualified types in macros (#5434) @RustedBones
- [gcp] Fix BQ BIGNUMERIC type support (#5428) @RustedBones
- Use hash join when writing sparkey (#5402) @aslotnick
📗 Documentation
- Prepare release v0.14.6 (#5437) @RustedBones
- doc: Update doc regarding avro version and logical-type support (#5435) @RustedBones
🏗️ Build Improvements
- Update sbt, sbt-dependency-tree to 1.10.1 (#5426) @scala-steward
- Update sbt-mdoc to 2.5.4 (#5427) @scala-steward
- Update sbt-bom to 1.0.14 (#5425) @scala-steward
- Use sbt-bom (#5414) @RustedBones
- Update sbt-scoverage to 2.1.0 (#5422) @scala-steward
- Update scalafmt-core to 3.8.2 (#5399) @scala-steward
- Derive artifact dependent versions (#5395) @RustedBones
🌱 Dependency Updates
- Update magnolify to 0.7.4 (#5432) @RustedBones
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.19.1 (#5424) @scala-steward
- Update elasticsearch-java to 8.14.3 (#5429) @scala-steward
- Update beam to 2.57.0 (#5413) @RustedBones
- Use default beam avro version (#5415) @RustedBones
- Update circe-core, circe-generic, ... to 0.14.9 (#5420) @scala-steward
- Update mysql-connector-j to 9.0.0 (#5419) @scala-steward
- Update neo4j-java-driver to 4.4.18 (#5412) @scala-steward
- Update scalatest to 3.2.19 (#5408) @scala-steward
- Update mssql-jdbc to 12.6.3.jre11 (#5403) @scala-steward
- Update parquet-avro, parquet-column, ... to 1.14.1 (#5405) @scala-steward
- Update testcontainers-scala-elasticsearch, ... to 0.41.4 (#5398) @scala-steward
- Update metrics-core to 4.2.26 (#5394) @scala-steward
Contributors to this release
Contributors
turb, RustedBones, and 2 other contributors
Assets 3
v0.14.5
What's changed
Includes Beam 2.56.0 support.
warning: This release transitively pulls newer avro and jackson versions. If you need to depend on previous versions, you'll need to pin them in your build.
🚀 Enhancements
- (#5296) Support Parquet predicates/projections in tests (#5309) @clairemcginty
- Add support for Zstd coders (#5321) @kellen
- [scio-core](feature) Sample SCollection with max weight (#5352) @RustedBones
- [scio-core](feature) Add readFiles and readFilesWithPath apis (#5350) @RustedBones
🐛 Bug Fixes
- Fix sparkey loading issue for unknown keys (#5387) @RustedBones
- (fix)[avro] Use correct DatumWriter constructor (#5371) @RustedBones
- fix FixQuery scalafix rule for macro-generated classes (#5356) @clairemcginty
📜 Scalafix Migrations
- Refactor FixAvroCoder case application (#5353) @clairemcginty
- Update FixAvroCoder to match transformed Avro SCollections (#5351) @clairemcginty
📗 Documentation
- Update release table for 0.14.5 (#5392) @RustedBones
- Prefer waitUndilDone over waitUntilFinish in test and examples (#5349) @RustedBones
🧪 Test Improvements
- Drop slow test tag (#5391) @RustedBones
🏗️ Build Improvements
- Update scalac-options to 0.1.5 (#5385) @scala-steward
- Fix assembly conflict (#5382) @RustedBones
- Update munit to 1.0.0 (#5378) @scala-steward
- [avro] Fix CI test for avro latest (#5372) @RustedBones
- Update sbt, sbt-dependency-tree to 1.10.0 (#5365) @scala-steward
- Update sbt-typelevel to 0.7.1 (#5366) @scala-steward
- Update socco-ng to 0.11.12 (#5368) @RustedBones
- Update sbt-scalafix to 0.12.1 (#5357) @scala-steward
- Update sbt-scoverage to 2.0.12 (#5363) @scala-steward
- (chore) Update to codecov-action (#5354) @RustedBones
- Enable mima for test artifacts (#5343) @RustedBones
- Update scalacheck to 1.18.0 (#5344) @scala-steward
🌱 Dependency Updates
- Sync flink artifact (#5393) @RustedBones
- Update elasticsearch-java to 7.17.21 (#5390) @RustedBones
- Update elasticsearch-java to 8.14.0 (#5388) @scala-steward
- Update neo4j-java-driver to 4.4.17 (#5384) @scala-steward
- Update algebra, cats-core, cats-kernel to 2.12.0 (#5383) @scala-steward
- Update magnolify to 0.7.3 (#5381) @RustedBones
- Update mssql-jdbc to 12.6.2.jre11 (#5380) @scala-steward
- Update jedis to 5.1.3 (#5376) @scala-steward
- Update shapeless to 2.3.12 (#5374) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.18.1 (#5369) @scala-steward
- Update scala-compiler, scala-library, ... to 2.13.14 (#5362) @scala-steward
- Update mysql-connector-j to 8.4.0 (#5359) @scala-steward
- Update beam to 2.56 (#5346) @RustedBones
- Update circe-core, circe-generic, ... to 0.14.7 (#5360) @scala-steward
Contributors to this release
Contributors
kellen, clairemcginty, and 2 other contributors
Assets 3
v0.14.4
Includes Beam 2.55.1 support.
🚀 Enhancements
- Robust throwable kryo coder (#5318) @RustedBones
- Use JacksonJsonpMapper as default for Elasticsearch (#5306) @kellen
- Allow String key type to transform SMB sources with CharSequence key (#5297) @clairemcginty
🐛 Bug Fixes
- [scio-avro] fix: allow conversions field in record (#5332) @RustedBones
- Add all avro logical type conversions to model (#5301) @RustedBones
- fix: safer implementation of distinctBy (#5299) @RustedBones
📜 Scalafix Migrations
- scalafix: Rule to migrate deprecated query to queryRaw (#5302) @RustedBones
📗 Documentation
- Prepare release v0.14.4 (#5342) @RustedBones
- [doc] Fix scaladoc link to class declared in package objects (#5316) @RustedBones
- Add staging dir to REPL docs (#5308) @kellen
🧪 Test Improvements
- Move test code in respective projects (#5330) @RustedBones
- [integration] Use literals for fromSchemaFile integration test (#5334) @RustedBones
- [integration] Fix avro integration test (#5333) @RustedBones
🏗️ Build Improvements
- Update sbt-site, sbt-site-paradox to 1.7.0 (#5324) @scala-steward
- Update sbt-paradox to 0.10.7 (#5325) @scala-steward
- Update scalafmt-core to 3.8.1 (#5320) @scala-steward
- Update sbt-buildinfo to 0.12.0 (#5310) @scala-steward
- Update sbt-assembly to 2.2.0 (#5304) @scala-steward
🌱 Dependency Updates
- Update scalacheck to 1.17.1 (#5341) @scala-steward
- Update cassandra-all to 3.11.17 (#5340) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.18.0 (#5339) @scala-steward
- Update pprint to 0.9.0 (#5336) @scala-steward
- Update scala-collection-compat to 2.12.0 (#5337) @scala-steward
- Update beam to 2.55.1 (#5322) @RustedBones
- Update elasticsearch-java to 8.13.2 (#5323) @scala-steward
- Update neo4j-java-driver to 4.4.15 (#5328) @scala-steward
- Update magnolify-avro, magnolify-bigtable, ... to 0.7.2 (#5319) @scala-steward
- Update elasticsearch v7 to 7.17.19 (#5317) @RustedBones
- Update beam to 2.55 (#5307) @RustedBones
- Update sbt-site, sbt-site-paradox to 1.6.0 (#5311) @scala-steward
- Update voyager to 2.0.6 (#5313) @scala-steward
Contributors to this release
Contributors
kellen, clairemcginty, and 2 other contributors
Assets 3
v0.14.3
🐛 Bug Fixes
- fix: Execute SmbIO output assertions (#5289) @RustedBones
- (fix #5290) Support empty input in TransformOverride.ofSource (#5293) @clairemcginty
- (Fix #5285) Allow String key type to read SMB sources written with CharSequence key type (#5291) @clairemcginty
🌱 Dependency Updates
- Update jedis to 5.1.2 (#5292) @scala-steward
- Update neo4j-java-driver to 4.4.14 (#5288) @scala-steward
🏗️ Build Improvements
- Update sbt-scalafix to 0.12.0 (#5287) @scala-steward
Contributors to this release
Contributors
clairemcginty, RustedBones, and scala-steward
Assets 3
v0.14.2
Contains a bugfix for duplicated SMB Transforms.
🐛 Bug Fixes
- (fix) Remove duplicated transform application (#5283) @RustedBones
🚀 Enhancements
- Make checkVersion more dynamic (#5282) @RustedBones
- Improve implicit coder not found message (#5281) @RustedBones
📗Documentation
- (doc) fix 0.14 migration guide broken links (#5278) @RustedBones
🏗️ Build Improvements
- Update scala to 2.13.13 and 2.12.19 (#5276) @RustedBones
Contributors to this release
Contributors
RustedBones
Assets 3
v0.14.1
Includes Beam 2.54.0 support.
🚀 Enhancements
- Add layer for low priority coder conflict (#5274) @RustedBones
- Add support of BIGNUMERIC in BigQueryIO (#5225) @shnapz
- Change dataflow runner check for parquet splittable do fn (#5264) @RustedBones
- Make sure Coder.gen produces informative error (#5258) @RustedBones
- Add default coder for ResourceId (#5244) @clairemcginty
- Add type bound for parquet-avro SCollection ops (#5230) @RustedBones
🐛 Bug Fixes
- (fix) Update SmbIO to support absolute path in test (#5277) @RustedBones
- Simplify BucketedInput serialization (#5270) @clairemcginty
- Fix byte[] equality issue for MockByteArraySparkeyReader (#5262) @clairemcginty
- Fix scio-repl assembly after bream 2.54 upgrade (#5263) @RustedBon
- Always set Parquet-Avro projection (#5234) @clairemcginty
- Fix SmbIO's testId naming (#5228) @clairemcginty
- Sort Path IDs in SmbIO (#5242) @clairemcginty
- Fix SmbIO side input support (fix #5240) (#5241) @clairemcginty
📜 Scalafix Migrations
- Make FixLogicalTypeSupplier more permissive (#5233) @clairemcginty
- Fix MatchError for FixAvroCoder (#5269) @clairemcginty
- Broaden use cases to FixAvroCoder (#5267) @clairemcginty
- Update FixAvroCoder to include SMB Avro reads (#5245) @clairemcginty
- Fix Scalafix rule for Avro package import (#5239) @clairemcginty
- Add Scalafix rule for SMB CharSequence key (#5236) @clairemcginty
- Scalafix: Add Avro coder import for SpecificRecord JobTest IOs (#5237) @clairemcginty
- Check options in scalafix (#5231) @kellen
- Inline scalafix (#5229) @kellen
📗 Documentation
- Update copyright year for scio site (#5261) @RustedBones
🧪 Test Improvements
- Fix populate it data (#5260) @RustedBones
- Add SortMergeBucketParityIT case for sortMergeTransform (#5224) @clairemcginty
🏗️ Build Improvements
- Update sbt-typelevel to 0.6.7 (#5275) @RustedBones
- Update sbt, sbt-dependency-tree to 1.9.9 (#5272) @scala-steward
- Update sbt-scoverage to 2.0.11 (#5273) @scala-steward
- Use released 0.14.0 scio version in scalafix (#5259) @RustedBones
- Update sbt-paradox-material-theme to 0.7.0 (#5251) @scala-steward
- Update scalafmt-core to 3.8.0 (#5254) @scala-steward
- Update scalactic to 3.2.18 (#5253) @scala-steward
- Update scalatest to 3.2.18 (#5255) @scala-steward
- Update sbt-scoverage to 2.0.10 (#5256) @scala-steward
- Cache TensorFlow Metadata proto files (#5246) @Duhemm
- Fix cache step for scalafix GHA (#5238) @RustedBones
- Bump release-drafter/release-drafter from 5 to 6 (#5226) @dependabot
🌱 Dependency Updates
- Update beam to 2.54.0 (#5235) @RustedBones
- Update elasticsearch-java to 8.12.2 (#5271) @scala-steward
- Update mssql-jdbc to 12.6.1.jre11 (#5268) @scala-steward
- Update cloud-sql-connector-jdbc-sqlserver, ... to 1.16.0 (#5249) @scala-steward
- Update testcontainers-scala-elasticsearch, ... to 0.41.3 (#5248) @scala-steward
Contributors to this release
@Duhemm, @RustedBones, @clairemcginty, @dependabot, @dependabot[bot], @kellen, @scala-steward and @shnapz
Contributors
kellen, shnapz, and 5 other contributors