Use default values for resource requests

[hategan]

Some scheduler configurations (e.g., Frontier@ORNL) require a node count. We do not mandate one, with the assumption that schedulers will have defaults and, if not, the (possibly misguided) spirit of PSI/J being a pass-through device which lets the scheduler decide how to handle a missing node count specification is maintained.

The problem exists when either a resource spec is missing and when the resource spec only specifies a process count (because the Slurm template does not use the computed counts).

This breaks our tests on Frontier. That, in itself, could be a statement that our tests are broken and should always specify a node count. More importantly, however, this breaks abstraction.

The point is that if we avoid defining defaults in PSI/J under the assumption that the scheduler will do the right thing with a missing value, that does not lead to uniform behavior, as evidenced above. Furthermore, the purpose of PSI/J is to clearly (and somewhat uniformly) define what a particular combination of values in the job spec, which, in this particular case, it fails to do. I would, therefore, argue that the meaning of a missing resource spec should be understood as "one process on one compute node".

The potential negative implication that I can think of is when some hypothetical scheduler might be configured to allocate fractional compute nodes when only a process count is specified, leading to an inability to specify such jobs on such schedulers, although when I have seen such scenarios, the scheduler tends to repurpose the notion of a node to mean the smallest fractional unit of a physical node.

This PR does two things:

• adds a default resource spec right before submission (in JobExecutor._check_job) and
• replaces instances of raw resource numbers in submit script templates with the corresponding computed values, which are always defined.

[hategan]

Merging for now because all tests pass. When @andre-merzky is back, we can discuss more (see #497)