Properly handle difference between restoring only model weights and full training state

[joeloskarsson]

The --restore_opt argument is supposed to differentiate between restoring only the model weights, and restoring the full training session, including optimizer state (this is the crucial one) and epoch number etc. This is currently implemented as a bit of a hack:

neural-lam/neural_lam/models/ar_model.py

Lines 759 to 761 in 43524f7

	if not self.restore_opt:
	opt = self.configure_optimizers()
	checkpoint["optimizer_states"] = [opt.state_dict()]

that can easily start causing problems as we build on it.

A proper way to handle this in Lightning is to differentiate between instantiating the model with load_from_checkpoint and calling Trainer.fit with ckpt_path. An example implementation that could be used is given here: joeloskarsson@e7d11c9