Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add language allowing TCK modification in service release after chall… #1496

Open
wants to merge 3 commits into
base: src
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion content/committees/specification/tckprocess/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,29 @@ update and release of an official distribution of the TCK including the
new exclude list. The associated `challenge` issue MUST be closed with
an `accepted` label to indicate it has been resolved.

The specification project may approve (user) workarounds for an `accepted` TCK challenge (as alternative to excluding TCK tests).
The specification project may approve (user) workarounds for an `accepted` TCK challenge (as an alternative to excluding TCK tests).

As another alternative to excluding a challenged test, it may be possible
to adjust the test validation logic to "expand" the validation check.
E.g. if a test expects a value "A1" for a specific variable "x", but a challenge
is raised arguing that the specification language actually allows for either
values "A1" and "A2" (but no other values) to be valid values for "x", then
it could be a valid course of action to modify the challenged test to allow
either "A1" OR "A2" for "x".

Since this line of thinking might be applied to cases that aren't quite as
straightfoward as this example, care should be taken when using this approach.
A particular danger is that an implementation that has already demonstrated compliance
before the challenge was raised might actually not pass the new, modified test.

To limit the confusion and additional work such a scenario would cause, if
there is already at least one certified compatible implementation before the challenge,
the new, modified TCK should be run against at least one such implementation (and ideally all of them)
before the changes are published, released, and finalized.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+100 for limiting confusion + additional work but I don't see how this can currently be coordinated in a way that doesn't cause other problems.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we could use a specific github issue for communicating that there is a staged (or possibly later on a release milestone released to a Maven repos) that can be used to validate that a TCK change is compatible with already certificated as compatible releases.


If such a change were released, and it was later found to cause a previously-certified implementation
to fail the new, modified test, then excluding the test would likely be the only option, and this would
require yet another, additional service release.


#### Rejected Challenges and Remedy {#_rejected_challenges_and_remedy}
Expand Down