fix typed BQ examples build, #2272 (#2284)

* exclude typed BQ examples from vanilla build * update build docs, #2272
spotify · Oct 3, 2019 · 01bece4 · 01bece4
1 parent 5b7c95e
commit 01bece4
Show file tree

Hide file tree

Showing 4 changed files with 21 additions and 13 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -27,13 +27,7 @@ You can also run `~mdoc` to automatically build the documentation on save.
 
 # Building locally
 
-Some example and test code in Scio depends on Google Cloud Platform access for BigQuery schemas, Cloud Dataflow service, etc. To build Scio locally without a GCP credential, first run the following script to generate BigQuery schema cache.
-
-```bash
-./scripts/gen_schemas.sh
-```
-
-When opening the project in IntelliJ IDEA, tick "Use sbt shell:" both "for imports" and "for builds".
+See [this page](https://spotify.github.io/scio/dev/build.html) for building Scio locally.
 
 # Code of Conduct
 

diff --git a/build.sbt b/build.sbt
@@ -817,7 +817,7 @@ lazy val scioExamples: Project = Project(
       if (BuildCredentials.exists) {
         HiddenFileFilter
       } else {
-        HiddenFileFilter || "TypedBigQueryTornadoes*.scala"
+        HiddenFileFilter || "TypedBigQueryTornadoes*.scala" || "TypedStorageBigQueryTornadoes*.scala"
       }
     },
     sources in doc in Compile := List()

diff --git a/site/src/paradox/Getting-Started.md b/site/src/paradox/Getting-Started.md
@@ -51,7 +51,7 @@ Unlike Hadoop, Scio or Dataflow input should be file patterns and not directorie
 Use the @javadoc[`DataflowRunner`](org.apache.beam.runners.dataflow.DataflowRunner) to execute pipelines on Google Cloud Dataflow service using managed resources in the Google Cloud Platform.
 
 ```
-neville@localhost scio $ sbt -Dbigquery.project=<BILLING_PROJECT>
+neville@localhost scio $ sbt
 [info] ...
 > project scio-examples
 [info] ...

diff --git a/site/src/paradox/dev/build.md b/site/src/paradox/dev/build.md
@@ -8,14 +8,28 @@ git clone https://github.com/spotify/scio.git
 
 ## Compiling
 
-Copy the pre generated BigQuery schemas:
+Build and test the code.
 
 ```bash
-cp -r scripts/bigquery .bigquery
+cd scio
+sbt test
 ```
 
-Define `bigquery.project` as a system property. You can use any value since we will be using the pre generated schemas.
+Some examples depend on Google Cloud Platform and are excluded by default if GCP credentials are missing. To enable them, authenticate yourself for GCP, set up default credentials and restart sbt.
 
 ```bash
-sbt -Dbigquery.project=dummy compile
+gcloud auth application-default login
+sbt test
 ```
+
+Alternatively you can populate pre-generated cache for BigQuery schemas to bypass GCP access.
+Define `bigquery.project` as a system property. The value can by anything since we'll hit cache instead.
+
+```bash
+./scripts/gen_schemas.sh
+sbt -Dbigquery.project=dummy-project test
+```
+
+## IntelliJ IDEA
+
+When opening the project in IntelliJ IDEA, tick "Use sbt shell:" both "for imports" and "for builds".