MLCOMPUTE-967 | changes for Spark drivers on k8s #3679

CaptainSame · 2023-08-17T13:51:33Z

Add newly created iam role to system paasta configs
Add methods to fetch the iam role
Use the iam role for creating the service account to start the Spark drivers with
Mount kubeconfig file to Spark driver pods
Point to the mounted kubeconfig using env variable
consolidate tron_tools, spark_run and service_configuration_lib methods

paasta_tools/tron_tools.py

CaptainSame · 2023-08-17T14:59:38Z

paasta_tools/tron_tools.py

                namespace=EXECUTOR_TYPE_TO_NAMESPACE[executor],
-                k8s_role=_spark_k8s_role(),
+                # k8s_role=_spark_k8s_role(),


Point env variable to KUBECONFIG = /etc/kubernetes/spark.conf (being used in adhoc mode)

Puppet kubeconfig to k8s agents to be available in pods

Mount KUBECONFIG to pods (extra volumes)

command: env include unset env variables for unsetting the env variables set by /etc/boto_cfg
e.g. --unset=AWS_ACCESS_KEY_ID --unset=AWS_SECRET_ACCESS_KEY

paasta_tools/tron_tools.py

…E-967_spark_drivers_on_k8s

88manpreet · 2023-08-23T18:43:34Z

Can you add description, testing, any impact on existing infra, rollout and rollback plan if needed?

paasta_tools/tron_tools.py

…E-967_spark_drivers_on_k8s

nemacysts · 2023-09-15T14:25:18Z

paasta_tools/tron_tools.py

+        if "extra_volumes" in result:
+            result["extra_volumes"] = result["extra_volumes"] + format_volumes(action_config.get_extra_volumes())
+        else:
+            result["extra_volumes"] = format_volumes(action_config.get_extra_volumes())


what are your thoughts on having TronActionConfig provide an override for get_extra_volumes() and doing the same if self.get_executor() == "spark": trick we do elsewhere to transparently add this mount? that we we can avoid doing the merge here which (without additional comments) could potentially confuse future yelpers reading this code

paasta_tools/tron_tools.py

…E-967_spark_drivers_on_k8s

…namically

… tools

…om/Yelp/paasta into MLCOMPUTE-967_spark_drivers_on_k8s

…E-967_spark_drivers_on_k8s

88manpreet

Changes lgtm once mypy tests pass.
Once someone from compute-infra reviews, should be good to go.

paasta_tools/cli/cmds/spark_run.py

paasta_tools/cli/schemas/tron_schema.json

paasta_tools/tron_tools.py

tests/test_utils.py

…E-967_spark_drivers_on_k8s

paasta_tools/spark_tools.py

paasta_tools/tron_tools.py

paasta_tools/cli/schemas/tron_schema.json

nemacysts · 2024-03-25T16:59:33Z

paasta_tools/tron_tools.py

+                "pool", load_system_paasta_config().get_tron_default_pool_override()
+            )
+            if not self.get_executor() == "spark"
+            else spark_tools.SPARK_DRIVER_POOL


fyi: we'll probably need to make this configurable in soaconfigs in the near future to allow for gpu drivers

paasta_tools/utils.py

…E-967_spark_drivers_on_k8s

nemacysts

once eman has also shipped, we can pair to pin paasta and release this :)

(might also be good to draft some comms for spark users at yelp in case they notice anything weird)

nemacysts · 2024-03-26T19:23:42Z

paasta_tools/utils.py

+    realized_cluster = (
+        load_system_paasta_config().get_eks_cluster_aliases().get(cluster, cluster)
+    )


i look forward to the day this is no longer necessary :)

paasta_tools/cli/schemas/tron_schema.json

paasta_tools/tron_tools.py

MLCOMPUTE-967 | add default iam role for Spark drivers on k8s

55682bd

CaptainSame requested review from nemacysts and 88manpreet August 17, 2023 13:51

CaptainSame commented Aug 17, 2023

View reviewed changes

paasta_tools/tron_tools.py Outdated Show resolved Hide resolved

CaptainSame commented Aug 17, 2023

View reviewed changes

nemacysts reviewed Aug 21, 2023

View reviewed changes

paasta_tools/tron_tools.py Outdated Show resolved Hide resolved

paasta_tools/tron_tools.py Outdated Show resolved Hide resolved

paasta_tools/tron_tools.py Outdated Show resolved Hide resolved

paasta_tools/tron_tools.py Outdated Show resolved Hide resolved

Sameer Sharma added 3 commits August 22, 2023 18:22

MLCOMPUTE-967 | fixed naming, added TODO

e9e89b0

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

003f0ae

…E-967_spark_drivers_on_k8s

MLCOMPUTE-967 | move default to SystemPaastaConfig

73c76fa

88manpreet reviewed Aug 23, 2023

View reviewed changes

paasta_tools/tron_tools.py Show resolved Hide resolved

Sameer Sharma added 4 commits September 11, 2023 16:46

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

34e3db1

…E-967_spark_drivers_on_k8s

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

1d61a56

…E-967_spark_drivers_on_k8s

MLCOMPUTE-967 | add volume mounts for kubeconfig

de6926a

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

7c98acd

…E-967_spark_drivers_on_k8s

nemacysts reviewed Sep 15, 2023

View reviewed changes

Sameer Sharma added 14 commits September 19, 2023 08:02

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

95bdac3

…E-967_spark_drivers_on_k8s

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

119f912

…E-967_spark_drivers_on_k8s

MLCOMPUTE-967 | test fixes

c895889

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

65ce515

…E-967_spark_drivers_on_k8s

MLCOMPUTE-967 | add spark to the list of executor types

6d7e16f

MLCOMPUTE-967 | use correct k8s url as spark master argument

c40a93d

MLCOMPUTE-967 | fix tests

ecde517

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

759a013

…E-967_spark_drivers_on_k8s

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

c28b60f

…E-967_spark_drivers_on_k8s

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

86947b5

…E-967_spark_drivers_on_k8s

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

c0f50e5

…E-967_spark_drivers_on_k8s

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

42c518b

…E-967_spark_drivers_on_k8s

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

759ca32

…E-967_spark_drivers_on_k8s

MLCOMPUTE-1057 | add flow to create Spark executor service account dy…

921dfbb

…namically

chi-yelp added 4 commits March 7, 2024 12:02

Specify prometheus scrape path

35938e4

Merge branch 'master' into MLCOMPUTE-967_spark_drivers_on_k8s

1960bff

Merge branch 'master' into MLCOMPUTE-967_spark_drivers_on_k8s

33f89cd

Update driver pod annotations for metrics scraping

88c5320

edingroot force-pushed the MLCOMPUTE-967_spark_drivers_on_k8s branch from e266651 to 88c5320 Compare March 10, 2024 10:29

Sameer Sharma added 5 commits March 14, 2024 13:00

MLCOMPUTE-967 | deprecate obsolete args, add mrjob capability to tron…

dca9d43

… tools

Merge branch 'MLCOMPUTE-967_spark_drivers_on_k8s' of https://github.c…

02b4ffc

…om/Yelp/paasta into MLCOMPUTE-967_spark_drivers_on_k8s

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

a17c3fa

…E-967_spark_drivers_on_k8s

MLCOMPUTE-967 | fix tests

275dbf3

MLCOMPUTE-967 | formatting

59622de

88manpreet self-requested a review March 14, 2024 14:46

88manpreet approved these changes Mar 14, 2024

View reviewed changes

MLCOMPUTE-967 | fix tests

97ea384

nemacysts reviewed Mar 14, 2024

View reviewed changes

Sameer Sharma added 3 commits March 25, 2024 14:29

MLCOMPUTE-967 | remove obsolete args, sanitise code

b4b5fb9

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

232307b

…E-967_spark_drivers_on_k8s

MLCOMPUTE-967 | formatting

e60d5bc

nemacysts reviewed Mar 25, 2024

View reviewed changes

Sameer Sharma added 4 commits March 25, 2024 17:14

MLCOMPUTE-967 | sanitise code

0da84e2

Merge branch 'master' of https://github.com/Yelp/paasta into MLCOMPUT…

39728a0

…E-967_spark_drivers_on_k8s

MLCOMPUTE-967 | fix tests

b409bb1

MLCOMPUTE-967 | fix tests

f430254

nemacysts requested a review from EmanElsaban March 26, 2024 19:13

nemacysts approved these changes Mar 26, 2024

View reviewed changes

EmanElsaban reviewed Mar 27, 2024

View reviewed changes

paasta_tools/cli/schemas/tron_schema.json Outdated Show resolved Hide resolved

paasta_tools/tron_tools.py Show resolved Hide resolved

paasta_tools/tron_tools.py Show resolved Hide resolved

MLCOMPUTE-967 | use already defined time_delta

c49d25d

EmanElsaban approved these changes Mar 27, 2024

View reviewed changes

CaptainSame merged commit 7135849 into master Mar 27, 2024
9 checks passed

edingroot mentioned this pull request Apr 12, 2024

Fix volume mount of spark-run on local cluster #3827

Merged

nurdann mentioned this pull request Apr 12, 2024

Support pod identity in spark-run #3828

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MLCOMPUTE-967 | changes for Spark drivers on k8s #3679

MLCOMPUTE-967 | changes for Spark drivers on k8s #3679

CaptainSame commented Aug 17, 2023 •

edited

Loading

CaptainSame Aug 17, 2023

88manpreet commented Aug 23, 2023

nemacysts Sep 15, 2023

88manpreet left a comment

nemacysts Mar 25, 2024

nemacysts left a comment •

edited

Loading

nemacysts Mar 26, 2024

MLCOMPUTE-967 | changes for Spark drivers on k8s #3679

MLCOMPUTE-967 | changes for Spark drivers on k8s #3679

Conversation

CaptainSame commented Aug 17, 2023 • edited Loading

CaptainSame Aug 17, 2023

Choose a reason for hiding this comment

88manpreet commented Aug 23, 2023

nemacysts Sep 15, 2023

Choose a reason for hiding this comment

88manpreet left a comment

Choose a reason for hiding this comment

nemacysts Mar 25, 2024

Choose a reason for hiding this comment

nemacysts left a comment • edited Loading

Choose a reason for hiding this comment

nemacysts Mar 26, 2024

Choose a reason for hiding this comment

CaptainSame commented Aug 17, 2023 •

edited

Loading

nemacysts left a comment •

edited

Loading