[SPARK-34795][SQL][TESTS] Adds a new job in GitHub Actions to check the output of TPC-DS queries #31886

maropu · 2021-03-18T22:50:13Z

What changes were proposed in this pull request?

This PR proposes to add a new job in GitHub Actions to check the output of TPC-DS queries.

NOTE: I've checked that the new job took 17m 35s in the GitHub Actions env.

Why are the changes needed?

There are some cases where we noticed runtime-realted bugs after merging commits (e.g. .SPARK-33822). Therefore, I think it is worth adding a new job in GitHub Actions to check query output of TPC-DS (sf=1).

Does this PR introduce any user-facing change?

No.

How was this patch tested?

The new test added.

.github/workflows/build_and_test.yml

maropu · 2021-03-18T22:54:52Z

sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala

@@ -405,7 +382,7 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession with SQLHelper
      val goldenOutput = {
        s"-- Automatically generated by ${getClass.getSimpleName}\n" +
        s"-- Number of queries: ${outputs.size}\n\n\n" +
-        outputs.zipWithIndex.map{case (qr, i) => qr.toString}.mkString("\n\n\n") + "\n"
+        outputs.mkString("\n\n\n") + "\n"


I just removed the unnecessary code here.

sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestHelper.scala

maropu · 2021-03-18T23:14:17Z

NOTE: I saw the Xiao's comment in the recent release process and then I made this PR.

SparkQA · 2021-03-18T23:58:54Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40809/

dongjoon-hyun · 2021-03-19T00:12:31Z

cc @wangyum , @HyukjinKwon because @wangyum made this attempt already before.

[SPARK-34211][SQL][TESTS] Benchmark TPC-DS with 1GB scale factor #31303

SparkQA · 2021-03-19T00:26:02Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40809/

maropu · 2021-03-19T00:28:47Z

Yea, but this PR is not for the benchmark purpose.

HyukjinKwon · 2021-03-19T00:55:00Z

I prefer to have this in GA ... cc @wangyum who should be the best person to review.
cc @cloud-fan too

cloud-fan · 2021-03-19T03:36:38Z

+1 to have this check!

sql/core/src/test/resources/tpcds-query-results/v1_4/q1.sql.out

maropu · 2021-03-21T03:33:58Z

NOTE: I'm checking if the result in the golden files is correct query-by-query.

SparkQA · 2021-03-21T04:42:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40878/

SparkQA · 2021-03-21T04:51:21Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40878/

SparkQA · 2021-03-21T08:23:26Z

Test build #136296 has finished for PR 31886 at commit 4185e73.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-03-21T12:51:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40889/

SparkQA · 2021-03-21T13:01:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40889/

SparkQA · 2021-03-21T16:19:00Z

Test build #136307 has finished for PR 31886 at commit 582c480.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-03-22T05:32:53Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40905/

SparkQA · 2021-03-22T08:36:36Z

Test build #136323 has finished for PR 31886 at commit 7738415.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2021-03-23T07:09:06Z

sql/core/src/test/resources/tpcds-query-results/v1_4/q17.sql.out

+-- !query schema
+struct<i_item_id:string,i_item_desc:string,s_state:string,store_sales_quantitycount:bigint,store_sales_quantityave:double,store_sales_quantitystdev:double,store_sales_quantitycov:double,as_store_returns_quantitycount:bigint,as_store_returns_quantityave:double,as_store_returns_quantitystdev:double,store_returns_quantitycov:double,catalog_sales_quantitycount:bigint,catalog_sales_quantityave:double,catalog_sales_quantitystdev:double,catalog_sales_quantitycov:double>
+-- !query output
+


This should be

+AAAAAAAAKPFEAAAA Recently right TN 1 99.0 NULL NULL 1 66.0 NULL NULL 1 32.0 NULL NULL

This seems to be caused by the wrong schema of date_dim: https://github.com/apache/spark/pull/31012/files#r599447123

Fixed in #31012

maropu · 2021-03-23T07:44:16Z

sql/core/src/test/resources/tpcds-query-results/v1_4/q41.sql.out

+-- !query schema
+struct<i_product_name:string>
+-- !query output
+


This should be:

ableationableought anticallyeingese callycallyeingese oughtationableought

A PR to fix this is: #31940

dongjoon-hyun

This PR seems to have two independent themes. Could you split this PR into two?

One is having build_and_test.yml change only.
One is the other test suite change and golden-file fix.

maropu · 2021-03-23T12:47:19Z

okay, @dongjoon-hyun ~ After I finish checking all the TPCDS query results, I'll do that (I'm still checking them)

SparkQA · 2021-03-23T14:37:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40987/

SparkQA · 2021-03-23T14:44:36Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40987/

SparkQA · 2021-03-23T18:59:11Z

Test build #136403 has finished for PR 31886 at commit 20f9e64.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-03-30T01:26:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41257/

SparkQA · 2021-03-30T01:57:35Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41257/

SparkQA · 2021-03-30T02:24:29Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41260/

SparkQA · 2021-03-30T02:30:02Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41260/

SparkQA · 2021-03-30T04:48:36Z

Test build #136675 has finished for PR 31886 at commit 2d891e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2021-03-30T05:43:38Z

Please wait me a few hours. I will verify the result first.

SparkQA · 2021-03-30T05:48:05Z

Test build #136678 has finished for PR 31886 at commit 0a19585.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-03-30T05:50:17Z

@wangyum, don't worry. There are few more issues to address (#31886 (comment)). It will take more.

HyukjinKwon

I think this is fine. I prefer to switch maropu/spark-tpcds-sf-1 to databricks/spark-sql-perf. I roughly checked it internally, and it seems fine to open source this repo in the future as well.

We could merge this one, and maybe @wangyum can follow-up to switch to databricks/spark-sql-perf? I will leave it to both of you to merge and how to proceed.

LGTM (pending #31886 (comment)) in any event.

SparkQA · 2021-03-30T13:22:55Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41297/

SparkQA · 2021-03-30T17:34:11Z

Test build #136715 has finished for PR 31886 at commit 841005c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2021-03-31T00:48:55Z

Thanks, @HyukjinKwon @wangyum . Merged to master.

maropu · 2021-03-31T00:50:22Z

If someone finds the failure of the new GA job, please let me know. cc: @dongjoon-hyun @cloud-fan @viirya @gatorsmile

maropu · 2021-03-31T00:51:50Z

After we resolve the pending issues on databricks/spark-sql-perf, I'm planning to add this job into branch-3.1./3.0, too.

HyukjinKwon · 2021-03-31T01:19:26Z

Thanks @maropu.

viirya · 2021-03-31T02:18:16Z

Thanks @maropu

gatorsmile · 2021-03-31T06:17:40Z

This is awesome! We should do it 5 years ago. :-)

… Adds a new job in GitHub Actions to check the output of TPC-DS queries ### What changes were proposed in this pull request? This PR proposes to add a new job in GitHub Actions to check the output of TPC-DS queries. NOTE: To generate TPC-DS table data in GA jobs, this PR includes generator code implemented in #32243 and #32460. This is the backport PR of #31886. ### Why are the changes needed? There are some cases where we noticed runtime-realted bugs after merging commits (e.g. .SPARK-33822). Therefore, I think it is worth adding a new job in GitHub Actions to check query output of TPC-DS (sf=1). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The new test added. Closes #32462 from maropu/TPCDSQUeryTestSuite-Branch3.1. Authored-by: Takeshi Yamamuro <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

… Adds a new job in GitHub Actions to check the output of TPC-DS queries ### What changes were proposed in this pull request? This PR proposes to add a new job in GitHub Actions to check the output of TPC-DS queries. NOTE: To generate TPC-DS table data in GA jobs, this PR includes generator code implemented in #32243 and #32460. This is the backport PR of #31886. ### Why are the changes needed? There are some cases where we noticed runtime-realted bugs after merging commits (e.g. .SPARK-33822). Therefore, I think it is worth adding a new job in GitHub Actions to check query output of TPC-DS (sf=1). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The new test added. Closes #32479 from maropu/TPCDSQueryTestSuite-Branch3.0. Authored-by: Takeshi Yamamuro <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

… Adds a new job in GitHub Actions to check the output of TPC-DS queries ### What changes were proposed in this pull request? This PR proposes to add a new job in GitHub Actions to check the output of TPC-DS queries. NOTE: To generate TPC-DS table data in GA jobs, this PR includes generator code implemented in apache#32243 and apache#32460. This is the backport PR of apache#31886. ### Why are the changes needed? There are some cases where we noticed runtime-realted bugs after merging commits (e.g. .SPARK-33822). Therefore, I think it is worth adding a new job in GitHub Actions to check query output of TPC-DS (sf=1). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The new test added. Closes apache#32462 from maropu/TPCDSQUeryTestSuite-Branch3.1. Authored-by: Takeshi Yamamuro <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

maropu commented Mar 18, 2021

View reviewed changes

.github/workflows/build_and_test.yml Outdated Show resolved Hide resolved

maropu commented Mar 18, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestHelper.scala Show resolved Hide resolved

github-actions bot added INFRA SQL labels Mar 19, 2021

wangyum reviewed Mar 19, 2021

View reviewed changes

sql/core/src/test/resources/tpcds-query-results/v1_4/q1.sql.out Outdated Show resolved Hide resolved

maropu commented Mar 23, 2021

View reviewed changes

maropu mentioned this pull request Mar 23, 2021

[SPARK-34083][SQL] Using TPCDS original definitions for char/varchar columns #31012

Closed

dongjoon-hyun previously requested changes Mar 23, 2021

View reviewed changes

maropu mentioned this pull request Mar 30, 2021

Use CHAR/VARCHAR types in TPCDSTables databricks/spark-sql-perf#198

Open

review

0a19585

maropu force-pushed the TPCDSQueryTestSuite branch from 2d891e9 to 0a19585 Compare March 30, 2021 01:20

Use ref when checking out the repo

841005c

HyukjinKwon approved these changes Mar 30, 2021

View reviewed changes

wangyum approved these changes Mar 30, 2021

View reviewed changes

maropu closed this in 46f96e9 Mar 31, 2021

maropu mentioned this pull request Apr 6, 2021

[SPARK-34944][SQL][TESTS] Replace bigint with int for web_returns and store_returns in TPCDS tests to employ correct data type #32037

Closed

[SPARK-34795][SQL][TESTS] Adds a new job in GitHub Actions to check the output of TPC-DS queries #31886

[SPARK-34795][SQL][TESTS] Adds a new job in GitHub Actions to check the output of TPC-DS queries #31886

Conversation

maropu commented Mar 18, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

maropu Mar 18, 2021

Choose a reason for hiding this comment

maropu commented Mar 18, 2021

SparkQA commented Mar 18, 2021

dongjoon-hyun commented Mar 19, 2021

SparkQA commented Mar 19, 2021

maropu commented Mar 19, 2021

HyukjinKwon commented Mar 19, 2021

cloud-fan commented Mar 19, 2021

maropu commented Mar 21, 2021

SparkQA commented Mar 21, 2021

SparkQA commented Mar 21, 2021

SparkQA commented Mar 21, 2021

SparkQA commented Mar 21, 2021

SparkQA commented Mar 21, 2021

SparkQA commented Mar 21, 2021

SparkQA commented Mar 22, 2021

SparkQA commented Mar 22, 2021

maropu Mar 23, 2021 • edited Loading

Choose a reason for hiding this comment

maropu Mar 23, 2021

Choose a reason for hiding this comment

maropu Mar 24, 2021

Choose a reason for hiding this comment

maropu Mar 23, 2021

Choose a reason for hiding this comment

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

maropu commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 23, 2021

SparkQA commented Mar 30, 2021

SparkQA commented Mar 30, 2021

SparkQA commented Mar 30, 2021

SparkQA commented Mar 30, 2021

SparkQA commented Mar 30, 2021

wangyum commented Mar 30, 2021

SparkQA commented Mar 30, 2021

HyukjinKwon commented Mar 30, 2021

HyukjinKwon left a comment • edited Loading

Choose a reason for hiding this comment

SparkQA commented Mar 30, 2021

SparkQA commented Mar 30, 2021

maropu commented Mar 31, 2021

maropu commented Mar 31, 2021

maropu commented Mar 31, 2021

HyukjinKwon commented Mar 31, 2021

viirya commented Mar 31, 2021

gatorsmile commented Mar 31, 2021

maropu Mar 23, 2021 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

HyukjinKwon left a comment •

edited

Loading