LSTM predictions from ensembles of training initializations

[jmframe]

@Zacmce @aaraney

The functionality of running ensembles of trained weights from random initializations is demonstrated in this fork/branch: https://github.com/jmframe/lstm/tree/ensemble_preds which constrains the trained models to be similar in input and training period, then in this branch: https://github.com/jmframe/lstm/tree/ensemble_preds_general, which has no constraints for the training configuration.

Changes:

# chang many variables to dictionaries, including:
        self.lstm = {} # The models themselves
        self.h_t = {} # The ensemble member states (hidden)
        self.c_t = {} # he ensemble member states (cell)

# And BMI outputs, and other aspects to save ensemble results, rather than single values.

Many changes were made to wrap specific code blocks in this loop: for i_ens in range(self.N_ENS):, where all the code then is acting on dictionaries, rather than singe value variables.

This code also removed the scalar functionality that required Pickle files. Scalar values are now read in from YML files, as described in #51.

Moved:
setattr(self, 'land_surface_water__runoff_volume_flux', self.streamflow_cms) and setattr(self, 'land_surface_water__runoff_depth', self.surface_runoff_mm/1000.0)
to the new function ensemble_output(), because that is now the last function called in the update()

changing the config parser to take in lists from the train_cfg_file key, even if it is a single path.

Addition:
Add function: def ensemble_output(self):

Trained neuralydrology models: (https://github.com/jmframe/lstm/tree/ensemble_preds/trained_neuralhydrology_models) has three directories with trained weights. For this demonstration, the weights come from the same initialization, but are taken from different epochs in the training procedure.

A few lines of code to help with the configuration parser, ensuring that we have a list of trained models, even if we only want to run a single LSTM model: https://github.com/jmframe/lstm/blob/225c0291c23a1bb49ae4766653eb5735981c3919/lstm/bmi_lstm.py#L918

Removals:
deleted the unused function read_initial_states, which was a placeholder for the hot start capabilities. Can be added later when #49 is addressed.

This code also removed the scalar functionality that required Pickle files.

Notes:
There are some features that should span across the ensemble members, such as the static attributes, initial forcings, scalars, etc. So in some cases there is a logical statement to only proceed with an operation if it is the first (or last) member of the ensemble, then those features are set for the rest of the code: https://github.com/jmframe/lstm/blob/225c0291c23a1bb49ae4766653eb5735981c3919/lstm/bmi_lstm.py#L280 and https://github.com/jmframe/lstm/blob/225c0291c23a1bb49ae4766653eb5735981c3919/lstm/bmi_lstm.py#L383.

Demonstration:
https://github.com/jmframe/lstm/blob/ensemble_preds_general/notebooks/run_lstm_with_bmi_aorc.ipynb

tested functionality with Nextgen to product a reasonable hydrograph output: