QLearner failing to stabilize due to odd spikes in QTable

[bwliv]

These don't seem organic. Clearly, something funky is happening in those episodes with the spikes (you see the makings of stabilization early but then it goes haywire). There's some random phenomena or bug causing crazy spikes in the Q table that shouldn't happen once the table is stabilized, and they are clearly completely undoing the training that had been done so far (as you see the table re-stabilizing after this happens).

This explains why the q table is unstable and nonsensical. These spikes continued continuously even when I tried 1000 episodes - they don't go away, and reset the training process.

@shreyasj3006 @mariemayadi this might require your knowledge of the implementation you wrote. These are definitely nonsensical jumps. I've debugged to death - one major note is I temporarily set commission and the sell penalty to zero to mitigate these jumps - and I have isolated/analyzed the episodes where it's happening, but there's something funky going on that is preventing our q learner from working properly. that might explain why the qtables have always looked wrong.

visualizations.ipynb (pushing new version momentarily) has good visualization of this phenomena.

The jumps appear to happen across all states - here are the 10 biggest row changes for one of these weird episode jumps:

here's a visualization of the spikes - the smaller (but still significant) ones continue like this indefinitely (I tried 1000 episodes earlier to confirm this

Here is what 1k episodes looks like (ignore the incorrect title of the graph):

[mariemayadi]

Thanks. Keen on further discussing this. Meaning of peaks and creating a new issue for us to investigate from our front etc.

[bwliv]

Looks better in latest push; separated epsilon from alpha and reduced epsilon by .005% every day via epsilon_decay parameter. Continues decaying more and more through each episode.

Are we satisfied by this or does @shreyasj3006 feel epsilon and gamma shouldn't have been separated? (CC @mariemayadi)

this stabilized results (although at the cost of reduced performance, so epsilon and epsilon_decay might require some tuning); it looks like there are a few days where there are huge stock returns or losses, and when random action is taken on those days (through epsilon) it throws the qtable into disarray because a massive reward (whether positive or negative) is applied in a different place than usual.

i think this is OK. we can note on report.

definitely a bigger theme of the qtable being funky, that's still a struggle. but this is progress.

[bwliv]

so basically:

question: how long does it take qtable to stabilize?

answer: it never stabilizes, unless we decay epsilon (the percent chance of a random action) over time... if it wasn't for a few days that can throw the data into disarray, it would stabilize in the first 10-20 episodes

additional note: forcing it to stabilize leads to worse performance - it performed much better without epsilon decaying.... so maybe letting the qtable never stabilize (and show these spikes like above) is optimal?

[bwliv]

here's the state now - decaying epsilon hurt performance b/c it led to holding more cash... maybe we just skip decaying epsilon? @shreyasj3006 @mariemayadi curious to hear your answers