Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request] [Kernel] Add Timestamp-NTZ write support #4103

Open
scottsand-db opened this issue Jan 29, 2025 · 2 comments · May be fixed by #4208
Open

[Feature Request] [Kernel] Add Timestamp-NTZ write support #4103

scottsand-db opened this issue Jan 29, 2025 · 2 comments · May be fixed by #4208
Assignees
@anoopj

Comments

@scottsand-db
Copy link
Collaborator

scottsand-db commented Jan 29, 2025 •
edited
Loading

Today, Kernel has Timestamp-NTZ read support. See the protocol spec here. See the read support PR here.

This issue requests that we add Timestamp-NTZ write support. Please refer to the delta-spark implementation as needed.

A rough sketch would include:

  • Figureout from Delta-Spark on what triggers enabling the table features. Is it having a column with type timestampNtx or delta property
  • [ALREADY COMPLETE] Kernel: Add support for writing timestampNTZ in the ParquetFileWriter (it may already have it, can't recall now)
  • Update the Kernel TxnBuilderImpl and TxnImpl to upgrade the protocol appropriately and generate a new Protocol action before commit
@anoopj anoopj self-assigned this Feb 11, 2025
@anoopj
Copy link
Collaborator

anoopj commented Feb 11, 2025 •
edited
Loading

Figureout from Delta-Spark on what triggers enabling the table features. Is it having a column with type timestampNtx or delta property

This is purely done through the existence of timeStampNtz fields. See the relevant section of Delta protocol for more details.

[ALREADY COMPLETE] Kernel: Add support for writing timestampNTZ in the ParquetFileWriter (it may already have it, can't recall now)

Looks done as part of #2907

@anoopj
Copy link
Collaborator

anoopj commented Feb 25, 2025

The write support is mostly done by PR#4159.

Will add a few more tests as part of this issue.

@anoopj anoopj linked a pull request Mar 2, 2025 that will close this issue
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@anoopj anoopj

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Kernel] New tests for Timestamp_NTZ writes anoopj/delta
2 participants
@anoopj @scottsand-db