-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix time-series filtering #203
Conversation
To have a fixed version for 0.3.x for compatibility with Spark 3.2.x, the commits of this PR should be merged into a new 0.3.5 version release, and updated accordingly. We can start thinking about making a separate branch for version 0.3.x and 0.4.x / further versions. |
Codecov Report
@@ Coverage Diff @@
## main #203 +/- ##
==========================================
+ Coverage 93.81% 93.83% +0.01%
==========================================
Files 85 85
Lines 2087 2091 +4
Branches 175 172 -3
==========================================
+ Hits 1958 1962 +4
Misses 129 129
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Hello @alexeiakimov ! Could you review this quick fix? Thank you 🙌 |
Description
Fixes #202 (and #201 😃 )
Type of change
It is a Bug Fix for the reading part, it does not include breaking changes on the algorithm or the writing of the files.
Now
TimestampType
andDateType
are pre-processed onQueryFilterUtils
to return a Millisecond representation that can be filtered properly by the index.Checklist:
Here is the list of things you should do before submitting this pull request:
How Has This Been Tested? (Optional)
I've added a test on
src/test/scala/io/qbeast/spark/index/query
calledTimeSeriesQueryTest
. In this class, I've tested four different ways of creating an index on a Date / Timestamp column. Each one were failing differently, and should fail if executed with previous commits.Test Configuration:
TBD.