[SYSTEMDS-3656] Update Hadoop 3.3.6

[BACtaki]

No description provided.

[j143]

For Hadoop upgrade, we simply change the pom version.
But, we can have a look at whether any specific internal updates are required.

[Baunsgaard]

The change is fine, and most likely all good. But we need to test it with an actual Hadoop instance. The test is what makes this upgrade harder than it seems on the surface. It is the same for the Spark update.

[BACtaki]

The change is fine, and most likely all good. But we need to test it with an actual Hadoop instance. The test is what makes this upgrade harder than it seems on the surface. It is the same for the Spark update.

@Baunsgaard What would testing entail? Is it as simple as running all unit/integration tests on a Hadoop cluster?

[Baunsgaard]

The change is fine, and most likely all good. But we need to test it with an actual Hadoop instance. The test is what makes this upgrade harder than it seems on the surface. It is the same for the Spark update.

@Baunsgaard What would testing entail? Is it as simple as running all unit/integration tests on a Hadoop cluster?

Good question.

The unit tests we have does use HDFS per default but you would not see if it works or not when running the tests unless some of them suddenly crash.

The way I did it before was to have simple scripts that write and read matrices and frames from an HDFS cluster (our normal execution should default to it if HDFS is detected). This does not give a full guarantee everything works but does sufficiently verify support.

[Baunsgaard]

I have tested the Hadoop 3.3.6, and it works with just the update. Thanks for initiating this @BACtaki .

The test i performed was a spark mode job,

I had some issues with writing to HDFS with hybrid mode, and need to look further into it.
The hybrid mode before the update also does not work the way I thought it should, and it might be a regression from some other changes.

For now we merge this update, since we also updated our cluster with the newer version and the update is (as far as i tested) backwards compatible with SystemDS.