2025-02-27 kernel meeting notes

[zachschuermann]

Summary

0.7.0 released! 0.7.1 upcoming. delta-rs integration.

Pull requests/ongoing threads

• Change the discussion format, particularly the pull requests/ongoing threads section.
• CI to build nightly with external integrations and make sure FFI works
• Upcoming priorities for kernel
• create_one API
• col [NOT] IN API
• snapshot incremental API
• @hntd187 targeting CDF reads, need timestamp work

Attendees

@nicklan @sebastiantia @hntd187 @roeap @zachschuermann @mightyshazam

notes stream

• 0.7.0 released!
• after 0.7.1, zach back on write path, v2checkpoint reads landing, checkpoint writes upcoming
• delta-rs integration:
  • standalone Snapshot implementation in delta-rs which uses kernel for log replay and everything
  • tried to be aligned with existing with delta-rs Snapshot API (mostly aligned)
  • perf hit in some code paths with kernel since we don't do caching/incremental snapshot/etc.
  • how invasive would it be to enable caching in default engine?
  • we (to implement protocol) need multi-object store support. one object store maps to one bucket/account/etc.
  • need to be able to read from multiple URLs
  • FYI delta-rs doesn't do muti-object store support.
  • delta-rs plans to use default engine
  • plan to use datafusion abstraction (datafusion object store?) delta-rs already uses datafusion object store registry heavily
• robert: few PRs in kernel, working on closing them!
• nick: flagging possible issue around refreshing credentials. how does object store know that it should get new credential or need new credenntial? client checks TTL? object store has internal refresh mechanics. object store doen't have the ability to get new credential from old one.
• robert: opening issue in delta-rs for kernelization.