#957 Implement earliest and latest functions

[currantw]

Signed-off-by: currantw [email protected]

Description

Implements earliest and latest relative date time functions in PPL.

Related Issues

Resolves #957

Check List

• Updated documentation:
  • PPL-Example-Commands.md
  • ppl-datetime.md
  • ppl-where-command.md
• Implemented unit tests
  • TimeUtilsTest
  • SerializableTimeUdfTest
• Implemented tests for combination with other commands
  • FlintSparkPPLBuiltInDateTimeFunctionITSuite
• New added source code should include a copyright header
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[currantw]

These aggregation commands don't appear to have ever been implemented. They are not used, referenced, or documented anywhere outside the auto-generated Antlr modules, that I can find. Moreover, I think that their functionality (as best as I can guess from the name, since they aren't documented) can just be accomplished using the existing min and max functions?

[kenrickyap]

are there spark issues that refer to this syntax? if so should mention that they are no longer relevant

or does sql has these implemented? If so might want to create future issues to implement this as we are striving for feature parity.

[currantw]

are there spark issues that refer to this syntax? if so should mention that they are no longer relevant

Good idea. I searched and didn't find anything.

or does sql has these implemented? If so might want to create future issues to implement this as we are striving for feature parity.

OpenSearch SQL does not appear to have these implemented either, and I similarly wasn't able to find any GitHub issues that made mentioned of it.

@YANG-DB I assume it also makes sense to remove this syntax from OpenSearch SQL for parity? Should I raise an issue and create a (tiny) PR for this?

[currantw]

@YANG-DB I have raise a cleanup issue on OpenSearch SQL here. Would you be able to triage it and I will create a (trivial) pull request to resolve it?

[currantw]

Update: here is the pull request.

[penghuo]

in the case where earliest is interpreted as
timestamp >= now() - 1s, depend on when the now() is resolved, it may produce insonsistent result?

[currantw]

I don't believe so. Relative timestamp are based on calls to CurrentTimestamp (see org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala). If you look at the documentation for CurrentTimestamp, it says that:

Returns the current timestamp at the start of query evaluation. All calls of current_timestamp within the same query return the same value.

So now(), relative_timestamp("now"), earliest("now", timestamp), latest("now", timestamp) and current_timestamp should all return consistent results within the same query. I tested this out manually as well to verify (i.e. repeated calls to these methods within the same query return the same now timestamp, down to the millisecond).