2025-02-20 kernel meeting notes

[OussamaSaoudi]

Summary

Pull requests/ongoing threads

Attendees

@nicklan @OussamaSaoudi-db @mightyshazam @sebastiantia @scovich @hntd187 @rtyler

Leftovers

• Change the discussion format, particularly the pull requests/ongoing threads section.
• CI to build nightly with external integrations and make sure FFI works
• Upcoming priorities for kernel
• create_one API
• col [NOT] IN API
• snapshot incremental API

notes stream

• nick: Trying to get stuff in for 0.7 release. We want the feature flags PR to merge in
• tyler: Arrow feature flag is ready to review
• nick: Typically we re-request review if a PR is ready to look at again.
• rtyler: delta-rs is unblocked on releasing, delta kernel is no longer a blocker. However, the feature flag is still really helpful for delta-rs, so it'd be nice to get in for a release
• Nick: Expression transform is done, just need to merge.
• Stephen: With expression transform stuff being a big change, we should make sure to unblock partition pruning PR.
• nick: The partition pruning isn't a breaking change, it's just a perf optimization. So we don't need to block the 0.7 release. We could release it in a 0.7.1
• tyler: Regarding v2Checkpoint: Is this a table feature that a lot of people are using? I thought we were running away from v2
• nick: We're running towards v2 checkpoints
• ryan: v1 checkpoints are 2 flavors: 1) A giant parquet file, 2) multi-part checkpoint. 2 is bad because it causes data corruption. Retries in spark break the table. v2 checkpoint uses a manifest approach like iceberg. It's fine to write a big checkpoint that contains v2 checkpoint data
• tyler: Is this table feature enabled by default?
• Ryan: v2 checkpoint is the default for liquid tables that databricks creates. v2 is strictly better because it is safer, allows you to potentially reuse checkpoint parts (sidecar files)
• tyler: If v2 checkpoints came in to a release in the near future, delta-rs would likely change the approach to prioritize integrating kernel in delta-rs. @roeap is working on the log replay path integration.
• nick: Do you need read + write, or just read?
• tyler: read is the major functionality we need.
• Sebastian: Read support PRs are all in review
• nick: Read support is likely ready in the very near future. (A couple weeks)
• Sebastian: All of read support could be merged in a week's time.
• Nick: We can cut a release immediately for v2 checkpoint support.
• tyler: Liquid tables writing v2 checkpoints makes this more pressing. We can point to a kernel branch to develop replay and checkpoint support. Will prioritize what we have now for release, then handle extra checkpoint feature afterwards.
• Zach: Looking to get 0.7 ASAP. That's blocked on 1) A json reader perf fix 2) expression transforms 3) the arrow feature flags
• Ryan: Arrow changed behaviour, and the PR to fix it is going out in 0.7
• Ryan: The behaviour change is probably affecting delta-rs. We should check on hacks that might be working around this issue.
• Tyler: I might've hit this. Scribd was pulling "esoteric" parquet that couldn't be compacted or optimized. We worked around that problem.
• Ryan: There was an issue where a null field was coming back unexpectedly
• Tyler: The issue came up when re-releasing delta-rs/datafusion code.
• Ryan: If you find a case that returns 0 or empty string when a null should be there, bring that up because it's likely a result of arrow's issues.
• Stephen: When implementing map functionality and hitting weird null behaviour, this might be a result of arrow's issues.
• Ryan: It is actually part of the arrow spec since 2022. They admit that null masks are not always valid. It just happened that the parquet and json reader were being nicer.
• Stephen: didn't encounter it from readers since I was creating record batches manually.
• Ryan: Creating record batches expects this strange behaviour, so it flys under the radar.
• Nick: Everything we need for 0.7 is likely ready.
• Zach: prototype PR for json reader speedup is here: [POC][wip] improve json reading with concurrent requests #704
• Zach: Production PR is coming soon.
• nick: We'll discuss if we should block 0.7 on the json reader speedup. Likely we shouldn't block the release on that.
• Ryan: If the reader needs more time, then lets leave it for later.
• Zach: It can be a point release. Not a breaking change, so major release not needed.