Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to config Spark to use spline using confinguration file #851

Closed
zacayd opened this issue Dec 30, 2024 · 0 comments
Closed

How to config Spark to use spline using confinguration file #851

zacayd opened this issue Dec 30, 2024 · 0 comments

Comments

@zacayd
Copy link

zacayd commented Dec 30, 2024

Hi
How can i configure spark to use spline but no on a promt
not like

spark-shell \
  --packages za.co.absa.spline.agent.spark:spark-3.3-spline-agent-bundle_2.12:2.2.0 \
  --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
  --conf "spark.spline.lineageDispatcher.http.producer.url=http://10.0.19.4:8080/producer"  \
  --conf "spark.driver.extraJavaOptions=--add-opens java.base/sun.net.www.protocol.jar=ALL-UNNAMED" \
  --conf "spark.executor.extraJavaOptions=--add-opens java.base/sun.net.www.protocol.jar=ALL-UNNAMED"

I mean that once I put on config files of spark and from there i can run scala and py files and see lineage
thanks
Zacay

@cerveada cerveada added this to Spline Dec 30, 2024
@github-project-automation github-project-automation bot moved this to New in Spline Dec 30, 2024
@AbsaOSS AbsaOSS locked and limited conversation to collaborators Jan 2, 2025
@wajda wajda converted this issue into discussion #852 Jan 2, 2025
@github-project-automation github-project-automation bot moved this from New to Done in Spline Jan 2, 2025

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
None yet
Projects
Status: Done
Development

No branches or pull requests

1 participant