[SYSTEMDS-3655] Update Spark Version 3.5.0

[BACtaki]

No description provided.

[BACtaki]

[WIP] Fix build failures with new Spark version

[j143]

Hi Badrul, there seem to be some components missing from spark dependencies perspective.

we could either create a simple checklist or one migration doc to help with this upgrade!

---

• 1. Pre-Upgrade Checks:

  • Review the release notes: Carefully review the release notes for Spark 3.4 and 3.5.x, paying particular attention to breaking changes, deprecations, and new features.

    • https://spark.apache.org/docs/latest/migration-guide.html

  • Review previous upgrade commits: We have a rich history of previous upgrades to help with the extent of changeset.

    • bb7fd14

  • Compatibility checks: Ensure the application's dependencies (libraries, frameworks) are compatible with Spark 3.5. Some libraries may require upgrades or adjustments to work with the newer version.

    java/org/apache/sysds/runtime/compress/colgroup/APreAgg.java:[22,31] package org.apache.commons.lang does not exist

  • Build and test application locally: Rebuild and test your application locally with Spark 3.5 to identify any compilation errors or functionality regressions. Try with components you are familiar with as a starting point.

  • Review logging and monitoring: Update logging and monitoring configurations to work with Spark 3.5. I believe the log4j template needs update.

• 2. Specific Considerations:

  • SQL, Datasets, and DataFrames: potential breaking changes in SQL syntax, DataFrame behavior, and schema inference.
  • Spark API usage: Review your internal Spark API usage for any deprecations or new features available in 3.5.x.
    • https://spark.apache.org/docs/latest/api/java/index.html

• 3. Design Steps for Upgrade: (optional)
  In case, we want to support upgrade in the hot running system.

  • Gradual rollout: Consider a phased rollout of the upgrade to minimize risk. We could start by deploying to a staging environment for testing, then gradually roll out to production in batches. (if there is production instance of systemds)
  • Monitor closely: monitor application after the upgrade for any performance regressions, errors, or unexpected behavior.
  • Rollback plan: Have a rollback plan in place in case you encounter unexpected issues after the upgrade. This might involve reverting to Spark 3.3 or rolling back specific changes you made.

[Baunsgaard]

I have verified the update works, after i updated some other dependencies.
I ran it in a cluster on the new Hadoop version as well.
So far, no breaking elements, but also no obvious improvements in performance.