[CALCITE-6451] Improve Nullability Derivation for Intersect and Minus

[vbarua]

Issue: CALCITE-6451

[vbarua]

I'm not sure if this is the best place for these tests. I'm open to suggestions about where they should go.

[NobiGo]

Hi @vbarua , I find some test in https://github.com/apache/calcite/blob/main/core/src/test/java/org/apache/calcite/test/JdbcTest.java#L4525-L4526 . Looks it is a good please for this tests.

[vbarua]

This interacts interestingly/poorly with the IntersectToDistinctRule, which rewrites Intersects to Unions using something like:

LogicalIntersect(all=[false])
  LogicalProject(a=[$0], b=[$1], c=[$2])
    LogicalTableScan(table=[[CATALOG, TEMP, FOO]])
  LogicalProject(a=[$0], b=[$1], c=[$2])
    LogicalTableScan(table=[[CATALOG, TEMP, BAR]])

which becomes

LogicalProject(a=[$0], b=[$1], c=[$2])
  LogicalFilter(condition=[=($9, 2)])
    LogicalAggregate(group=[{0, 1, 2}], agg#0=[COUNT()])
      LogicalUnion(all=[true])
        LogicalAggregate(group=[{0, 1, 2}], agg#0=[COUNT()])
          LogicalProject(a=[$0], b=[$1], c=[$2])
            LogicalTableScan(table=[[CATALOG, TEMP, FOO]])
        LogicalAggregate(group=[{0, 1, 2}], agg#0=[COUNT()])
          LogicalProject(a=[$0], b=[$1], c=[$2])
            LogicalTableScan(table=[[CATALOG, TEMP, BAR]])

What's interesting about this is that while it returns the same results, it's actually somewhat lossy with regards to the nullability information because we can't infer as tight nullability bounds for the second form as we can the first. That is to say, if FOO.a is nullable but BAR.a is not nullable, then with the first form we can say that in the output a will not be nullable but this is not the case with the second form.

One way I could see to work around this would be to improve the IntersectToDistinctRule by including filtering information when possible. That is, when a column is not nullable in all Intersect branches, we add an IS NOT NULL filter to exclude nulls:

LogicalProject(a=[$0], b=[$1], c=[$2])
  LogicalFilter(condition=[=($9, 2)])
    LogicalAggregate(group=[{0, 1, 2}], agg#0=[COUNT()])
      LogicalUnion(all=[true])
        LogicalAggregate(group=[{0, 1, 2}], agg#0=[COUNT()])
          LogicalFilter(condition=[IS NOT NULL($0)] <-- Filter here excludes NULL columns
            LogicalProject(a=[$0], b=[$1], c=[$2])
              LogicalTableScan(table=[[CATALOG, TEMP, FOO]])
        LogicalAggregate(group=[{0, 1, 2}], agg#0=[COUNT()])
          LogicalProject(a=[$0], b=[$1], c=[$2])
            LogicalTableScan(table=[[CATALOG, TEMP, BAR]])

However that might also require additional work because I don't believe that Calcite can use the presence of an IS NOT NULL filter to change the output nullability.

The reason this is a problem now is that when the IntersectToDisctinctRule is applied that types of the RelNode going in and RelNode coming out no longer match.

[mihaibudiu]

I will edit the PR title to match the JIRA ticket title.
You will have to do the same for the commit message.

[mihaibudiu]

I think you will need to fix the IntersectToDistinct rule before we can merge this PR.

[vbarua]

I've pushed some changes that address the issue with the IntersectToDistinctRule. Unfortunately, it looks there's a different issue as well. The test in:

calcite/core/src/test/resources/sql/misc.iq

Lines 1654 to 1665 in f44ed0a

	# INTERSECT
	select deptno from "scott".emp
	intersect
	select deptno from "scott".dept;
	+--------+
	| DEPTNO |
	+--------+
	| 10 |
	| 20 |
	| 30 |
	+--------+
	(3 rows)

is failing now because the output type of the relational tree no longer matches the type of the validated SQL tree. I suspect I need to update the SQL validation logic as well to address that. I will continue to look at this when I have a chance.

[mihaibudiu]

I will convert this to a draft, since it's not yet ready

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 90 days if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.