Skip to content

Commit

Permalink
Add Spark client user metadata search example to spark-client.md (#7918)
Browse files Browse the repository at this point in the history
* Add a metadata search example to spark-client.md

* Fix numbering
  • Loading branch information
itaigilo authored Jun 26, 2024
1 parent 2e57415 commit f58ff0f
Showing 1 changed file with 22 additions and 0 deletions.
22 changes: 22 additions & 0 deletions docs/reference/spark-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,5 +117,27 @@ Include this assembled jar (an "Überjar") from S3, from
*/
```

1. Search by user metadata:

```scala
import io.treeverse.clients.LakeFSContext

val namespace = "s3://bucket/repo/path/"
val df = LakeFSContext.newDF(spark, namespace)

val key = "SomeKey"
val searchedValue = "val3"
df.select("key", "user_metadata")
.filter(_.getMap[String, String](1).toMap.get(s"X-Amz-Meta-${key}").getOrElse("") == searchedValue)
.show()
/* output example:
+---------+-----------------------------------------------------+
|key |user_metadata |
+---------+-----------------------------------------------------+
|file1.txt|{X-Amz-Meta-SomeKey -> val3, X-Amz-Meta-Tag -> blue} |
|file8.txt|{X-Amz-Meta-SomeKey -> val3, X-Amz-Meta-Tag -> green}|
+---------+-----------------------------------------------------+
*/
```

[s3a-assumed-role]: https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/assumed_roles.html#Configuring_Assumed_Roles

0 comments on commit f58ff0f

Please sign in to comment.