[SPARK-49984][CORE] Fix duplicate JVM options

[Kimahriman]

What changes were proposed in this pull request?

Fix how the JVM options are supplemented with Java module options and IPv6 options. Get and set based on the exact value of spark.*.extraJavaOptions, and don't use the helper config which combines both default and extra Java options.

Why are the changes needed?

The current approach using DRIVER_JAVA_OPTIONS and EXECUTOR_JAVA_OPTIONS loads both the spark.*.extraJavaOptions and spark.*.defaultJavaOptions, adds the "supplemental" options, and then resaves the result in spark.*.extraJavaOptions. This is because DRIVER_JAVA_OPTIONS is a config like:

  private[spark] val DRIVER_JAVA_OPTIONS =
    ConfigBuilder(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS)
      .withPrepended(SparkLauncher.DRIVER_DEFAULT_JAVA_OPTIONS)

The result is spark.*.defaultJavaOptions being added multiple times to spark.*.extraJavaOptions, and when the full options are actually pulled to launch a process, three copies of what was in the spark.*.defaultJavaOptions ends up being included on the command line.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manual testing

Before:

$ pyspark --conf spark.driver.defaultJavaOptions="-Dfoo=bar"
>>> spark.conf.get('spark.driver.extraJavaOptions')
'-Djava.net.preferIPv6Addresses=false -Dfoo=bar -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dfoo=bar'
>>> spark.conf.get('spark.driver.defaultJavaOptions')
'-Dfoo=bar'

After:

$ pyspark --conf spark.driver.defaultJavaOptions="-Dfoo=bar"
>>> spark.conf.get('spark.driver.extraJavaOptions')
'-Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-modules=jdk.incubator.vector --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true'
>>> spark.conf.get('spark.driver.defaJavultJavaOptions')
'-Dfoo=bar'

Was this patch authored or co-authored using generative AI tooling?

No

[Kimahriman]

@LuciferYang @dongjoon-hyun since you created these methods

[dongjoon-hyun]

Thank you. Could you add a test case, @Kimahriman ?

[LuciferYang]

+1, Agree with @dongjoon-hyun

[Kimahriman]

Didn't see any tests when these were added, but added a test to the SparkContextSuite