Hot start capabilities

[jmframe]

Short description explaining the high-level reason for the new issue.

Current behavior

There hasn't been development of hot start capabilities in the LSTM BMI file.
Here is a link to the configuration with no hot start keys, neither save states or starting from saved states: https://github.com/NOAA-OWP/lstm/blob/master/bmi_config_files/01022500_hourly_slope_mean_precip_temp.yml, note that the key initial_state has a value if zero, which is used in the BMI initialization function:

lstm/lstm/bmi_lstm.py

Line 311 in 44a16c1

if self.cfg_bmi['initial_state'] == 'zero':

.

Below I describe a demonstration with those capabilities.

Note that I don’t exactly know how hot starts will be handled with Nextgen in general, so all of my code here is just prototyping the method based on guesses of what might happen with Nextgen.

Expected behavior

The LSTM can start a simulation with states exported at a particular time step from a previous run

Steps to replicate behavior (include URLs)

1. Saving states:
   Now here is a demo configuration file with the new key value to indicate to the BMI that the states will be saved at timestep 200,000: https://github.com/jmframe/lstm/blob/hotstart/bmi_config_files/01022500_hourly_slope_mean_precip_temp_save_states.yml. Note that this time value for saving states is just the number of time steps, which is arbitrary, and will likely be replaced by some standard time based on the operational framework, also note that we are still starting from initial states of zero. The BMI has been updated to use this time value for saving states to indicate the export and naming of a state file: https://github.com/jmframe/lstm/blob/ef6d51eca41285053c6982431ecba57c7cf44e78/lstm/bmi_lstm.py#L303. In this demonstration, the states are saved via Pickle file (./saved_states/state_200000.pkl), which is a standard serialization package from Python, but the states can be exported in any format desired, including ascii. Now, here is a notebook that uses the BMI functionality to run the LSTM model with this config file and does the state saving operation: https://github.com/jmframe/lstm/blob/hotstart/notebooks/run_lstm_with_bmi_save_states.ipynb.

2. Hot start:
   Now here is a demo configuration file with a new key value: start_time and the initial_state value now set to be the path to the pickle file that was exported with the state saving demonstration above (./saved_states/state_200000.pkl): https://github.com/jmframe/lstm/blob/hotstart/bmi_config_files/01022500_hourly_slope_mean_precip_temp_load_states.yml. There is a demonstration update to the BMI initialization now that looks for this file name to load in the “hot” states: https://github.com/jmframe/lstm/blob/ef6d51eca41285053c6982431ecba57c7cf44e78/lstm/bmi_lstm.py#L271. Instead of starting from zero values, the states are just loaded in from the pickle file. Here is a demonstration notebook that runs the LSTM module with this BMI configuration: https://github.com/jmframe/lstm/blob/hotstart/notebooks/run_lstm_with_bmi_load_states_hot_start.ipynb. Notice that the figure in cell seven has LSTM values at zero for the first half of the specified time period, then at time period 200,000 the simulation has identical values to those in the notebook above in the “saving states” section.

Screenshots

Simulation saving states at time 200,000:

Simulation loading in states exported from the simulation above, starting at time 200,000: