Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix README for working code copy/paste in Spark shell and instructions #237

Merged
merged 2 commits into from
Nov 28, 2023
Merged

Fix README for working code copy/paste in Spark shell and instructions #237

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
528fd63
Fix README for working code copy/paste in Spark shell and instructions
cdelfosse Nov 27, 2023
3107ed6
Use camelCase in Scala code example
cdelfosse Nov 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 18 additions & 18 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,22 +89,22 @@ $SPARK_HOME/bin/spark-shell \
**Read** the **CSV** source file placed inside the project.

```scala
val csv_df = spark.read.format("csv")
.option("header", "true")
.option("inferSchema", "true")
.load("./src/test/resources/ecommerce100K_2019_Oct.csv")
val csvDF = spark.read.format("csv").
option("header", "true").
option("inferSchema", "true").
load("./src/test/resources/ecommerce100K_2019_Oct.csv")
```

Indexing the dataset by writing it into the **qbeast** format, specifying the columns to index.

```scala
val tmp_dir = "/tmp/qbeast-spark"
val tmpDir = "/tmp/qbeast-spark"

csv_df.write
.mode("overwrite")
.format("qbeast")
.option("columnsToIndex", "user_id,product_id")
.save(tmp_dir)
csvDF.write.
mode("overwrite").
format("qbeast").
option("columnsToIndex", "user_id,product_id").
save(tmpDir)
```

#### SQL Syntax.
Expand All @@ -129,20 +129,20 @@ spark.sql("INSERT INTO table student SELECT * FROM visitor_students")
Load the newly indexed dataset.

```scala
val qbeast_df =
spark
.read
.format("qbeast")
.load(tmp_dir)
val qbeastDF =
spark.
read.
format("qbeast").
load(tmpDir)
```

### 4. Examine the Query plan for sampling
**Sampling the data**, notice how the sampler is converted into filters and pushed down to the source!

```scala
qbeast_df.sample(0.1).explain(true)
qbeastDF.sample(0.1).explain(true)
```
Go to the [Quickstart](./docs/Quickstart.md) or [notebook](docs/sample_pushdown_demo.ipynb) for more details.
Go to the [Quickstart](./docs/Quickstart.md) or [notebook](docs/sampleopushdown_demo.ipynb) for more details.

### 5. Interact with the format

Expand All @@ -151,7 +151,7 @@ Get **insights** or execute **operations** to the data using the `QbeastTable` i
```scala
import io.qbeast.spark.QbeastTable

val qbeast_table = QbeastTable.forPath(spark, tmp_dir)
val qbeastTable = QbeastTable.forPath(spark, tmpDir)

qbeastTable.getIndexMetrics()

Expand Down
Loading