Static Action Value Throughout Episode

[TimThacker]

Hi Hantao,

I have been working on an application of andes_gym for improving primary frequency response as opposed to secondary frequency response presented in the examples. I have noticed that for each episode, despite the number of action instances I apply, the action value is static. I do not see any significant differences between the parameters used in the example and my application.

Did you ever encounter an issue like this, or do you have an idea about where I should be looking to resolve this?

Thank you.

[cuihantao]

For some reason my reply did not go through. Check if you modified arrays in place when you set the action values to ANDES. If so, please share the few lines of code for setting the action values.

[TimThacker]

Hanto,
I did not change how and when the action values were applied to ANDES. I have continued to investigate this issue as it seems to be broad and difficult to explain without further review. Here are several things I have noticed:

• There are no issues with getting a different action value for each action instance during the "learning starts"
• The action value quickly becomes the same for each action instance in each episode when learning begins
• It appears that the default action noise and parameter noise for DDPG is zero

I seem to have created a band-aid fix of sorts by adding mandatory exploration via the implementation of action noise, but I am still testing this to see if it will give the results I expect. Perhaps working with a small frequency deviation, the deviation is large enough for the neural networks to determine a change in action throughout the episode? I am unsure, but this seems like a reasonable conclusion.

Looking at the stable_baselines documentation, the OrnsteinUhlenbeckActionNoise takes ActionNoise as an input. DDPG was presented and is most often implemented with Ornstein Uhlenbeck Noise. If I am correct that the default action noise is zero, than perhaps the OH noise is also zero because of how stable_baselines is structured. I will update you when I have explored whether or not the lack of noise is the issue.

[cuihantao]

Check if arrays are modified correctly in-place. If so, please share the few lines of code for applying the action. Regards, Hantao Cui

…

On Sep 20, 2022 at 1:00 PM -0500, Tim Thacker ***@***.***>, wrote: Hi Hantao, I have been working on an application of andes_gym for improving primary frequency response as opposed to secondary frequency response presented in the examples. I have noticed that for each episode, despite the number of action instances I apply, the action value is static. I do not see any significant differences between the parameters used in the example and my application. Did you ever encounter an issue like this, or do you have an idea about where I should be looking to resolve this? Thank you. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>