Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation specifies Kubeflow jobs, but not SparkApplication #4098

Open
danc opened this issue Jan 29, 2025 · 2 comments
Open

Documentation specifies Kubeflow jobs, but not SparkApplication #4098

danc opened this issue Jan 29, 2025 · 2 comments

Comments

@danc
Copy link

danc commented Jan 29, 2025

Could doc clarify if Kubeflow SparkApplications are supported by kueue ?

I'm currently able to submit Jobs to kueue, but SparkApplications appear not to be passing via the LocalQueue (maybe some info missing)
Example failing yaml. (SparkApplication succeeds, but is not managed by Queue) :

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  generateName: spark-pi-ingestion-
  namespace: mining
  labels:
    app: spark
    kueue.x-k8s.io/queue-name: lq-spark
spec:
  timeToLiveSeconds: 60
  type: Scala
  mode: cluster
  image: "apache/spark:3.4.1"
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.4.1.jar"
  arguments:
    - "10"
  sparkVersion: "3.4.1"
  restartPolicy:
    type: Never
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.4.1
      app: spark
    serviceAccount: logpickr-api
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 3.4.1
      app: spark
@everpeace
Copy link

everpeace commented Jan 30, 2025

Could doc clarify if Kubeflow SparkApplications are supported by kueue ?

I think Tasks / Run Workloads / Kubeflow Jobs page clearly states so (the list does not contain Kubeflow Spark Operator(SparkApplication)).

And, Tasks / Developer Tools / Integrate a custom Job with Kueue also clearly instructs how to support custom jobs in Kueue.

By the way, Kueue community already aware the spark issue and already have grand design how to support spark workloards in Kueue: #1656

@danc
Copy link
Author

danc commented Jan 30, 2025

Thanks for the confirmation, shared links, and grand design perspective.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants