add conformance reporting

[yliuuuu]

Relevant Issues

• [Closes/Related To] Issue #XXX

Description

• < A detailed description > xxx

Other Information

• Updated Unreleased Section in CHANGELOG: [YES/NO]

  • < If NO, why? >

• Any backward-incompatible changes? [YES/NO]

  • < If YES, why? >
  • < For this purpose, we define backward-incompatible changes as changes that—when consumed—can potentially result in
    errors for users that are using our public APIs or the entities that have public visibility in our code-base. >

• Any new external dependencies? [YES/NO]

  • < If YES, which ones and why? >
  • < In addition, please also mention any other alternatives you've considered and the reason they've been discarded >

• Do your changes comply with the Contributing Guidelines
  and Code Style Guidelines? [YES/NO]

License Information

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[github-actions]

Conformance comparison report-Cross Engine

	Base legacy	eval	+/-
% Passing	92.54%	3.01%	-89.53%
✅ Passing	5384	175	-5209
❌ Failing	434	5644	5210
🔶 Ignored	0	0	0
Total Tests	5818	5819	1
Number passing in both: 175

Number failing in both: 434

Number passing in legacy engine but fail in eval engine: 5210

Number failing in legacy engine but pass in eval engine: 0
⁉️ CONFORMANCE REPORT REGRESSION DETECTED ⁉️

Conformance comparison report-Cross Commit-LEGACY

	Base (8321b9a)	4d90ef1	+/-
% Passing	92.54%	92.54%	0.00%
✅ Passing	5384	5384	0
❌ Failing	434	434	0
🔶 Ignored	0	0	0
Total Tests	5818	5818	0
Number passing in both: 5384

Number failing in both: 434

Number passing in Base (8321b9a) but now fail: 0

Number failing in Base (8321b9a) but now pass: 0

Conformance comparison report-Cross Commit-EVAL

	Base (8321b9a)	4d90ef1	+/-
% Passing	3.01%	3.01%	0.00%
✅ Passing	175	175	0
❌ Failing	5644	5644	0
🔶 Ignored	0	0	0
Total Tests	5819	5819	0
Number passing in both: 175

Number failing in both: 5644

Number passing in Base (8321b9a) but now fail: 1

Number failing in Base (8321b9a) but now pass: 1
⁉️ CONFORMANCE REPORT REGRESSION DETECTED ⁉️. The following test(s) were previously passing but now fail:

Click here to see

• path expression with ambiguous table alias (lowercase), compileOption: LEGACY

The following test(s) were previously failing but now pass. Before merging, confirm they are intended to pass:

Click here to see

• path expression with ambiguous table alias (lowercase), compileOption: LEGACY

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (eval-report-base@c7466e4). Click here to learn what that means.

❗ Current head 92642b6 differs from pull request most recent head 3d6a2be. Consider uploading reports for the commit 3d6a2be to get more accurate results

Additional details and impacted files

@@                 Coverage Diff                 @@
##             eval-report-base      #14   +/-   ##
===================================================
  Coverage                    ?   49.27%           
  Complexity                  ?     1046           
===================================================
  Files                       ?      166           
  Lines                       ?    13395           
  Branches                    ?     2504           
===================================================
  Hits                        ?     6600           
  Misses                      ?     6138           
  Partials                    ?      657           

Flag	Coverage Δ
CLI	11.86% <0.00%> (?)
EXAMPLES	80.28% <0.00%> (?)
LANG	54.71% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.