2023-08-31 kernel meeting notes

[zachschuermann]

summary

main threads:

• @wjones127 PR: feat: parallel sync and async APIs #25
• action item: get CI running against forks (@zachschuermann)
• action item: sync about merging @wjones127 PR

attendees

@tdas @rtyler @wjones127 @nicklan @MrPowers @ryan-johnson-databricks @zachschuermann

notes stream

threads

• Rust updates, todos, targets, etc. (hackathon and @wjones127 PR feat: parallel sync and async APIs #25)
• @MrPowers updates

Rust updates

Hackathon: duckDB at first, now just C API over arrow C data interface

• Tyler: curious about C header file
• Will: support arrow stream API as well? Yup.
• Zach: curious to discuss the tradeoff between C FFI and using other lang-specific packages e.g. Cxx or Maturin/PyO3, etc.

Will PR for sync interfaces

• Eliminates async from APIs but keeps it open for TableClient implementations
• Implemented such that you could share runtime between things
• Tyler: async being low in API? What is the benefit of async?
• read_file vs read_files function
• You can do parallel/concurrent storage requests
• read_parquet/_json can allow for concurrency
• From last meeting: point of async is the IO pieces - important to be available at the IO layer. Good example is ObjectStore crate/trait.

@ryan-johnson-databricks where in kernel proper are we sending batch files?

• If there is a batch of files that could be read in parallel we want to allow that
• Where is this spot?

@nicklan ideally kernel doesn’t care about sync/async

• Push execution out to the engine
• tokio/async lives outside kernel
• Still understanding whether it is possible/tractable

discussion on sync/async

• @ryan-johnson-databricks thoughts:
  • parallel file reads mean memory consumption: this is liability not asset
  • Hashtable during log replay is required - can’t get around it
  • Decision to read multiple files in parallel or not may indicate a flaw in delta-kernel abstraction
• clarify: Default client is async but interfaces are sync
• read_files iterator
• Can pass into client from them to do read-ahead
  • Do them in order but please run ahead if you are a smart engine
• note: Might be some work to bridge between rust/etc. (rust Stream [async iterator] is different than Iterator [sync])
• Run-ahead iterator would expose a sync interface since you really do have to read the next one next but could do background work etc.
  • Rust: lots of futures and you await all of them

Question from @nicklan to @wjones127: should we merge PR and iterate? Or iterate un-merged? Action item: follow-up

context: Tyler/Robert implemented streams/async (without runtime)

• Challenge: C FFI
• Ask for @nicklan/anyone can we investigate C FFI for async?
  • (wrap async around sync)
  • sync around async you can just block_on
• Tokio into default client? Not required/easy fix. Could just implement sync client.
• H-shaped API: two sides are the same engine stuff and kernel is line in the middle
  • I think both sides of H will be async - zach

other updates

• @MrPowers Trino
• @MrPowers Delta-lake community meetings: want to pick them up again.
  • Where, when, who
• Should do a meet-up!