How to shape the reward function based on multiple parameters?

Question

After going over this post about reward shaping, I still find it difficult to define it w.r.t to my specific problem. Suppose that I measure some state $s=\phi$, which is some angle. I also use actions $a=\tau$ that are motor torques. For example, given the state $\phi_0$, I would like to apply torque $\tau_0$, which will bring my system to a new angle state $\phi_1$. There are $2$ aspects that are important:

1. I aim to bound $\phi$ to some range $[\phi_{min},\phi_{max}]$
2. Large changes between two consecutive actions should be avoided , i.e $\forall t$ $||a_t-a_{t-1}||<\epsilon$ for some small $\epsilon>0$.

Given those, a Naive reward shaping would be:

1. if $\phi\notin[\phi_{min},\phi_{max}]:$ set $r\leftarrow r-\phi$
2. if $||a_t-a_{t-1}||\geq\epsilon:$ set $r\leftarrow r-\||a_t-a_{t-1}||$

The problem is that first, $\phi$ and $||a_t-a_{t-1}||$ may very well be on different scales. if for example $\phi\gg||a_t-a_{t-1}||$, the second line wouldn't affect the loss, and vice versa. Furthermore, under a more "physical" scope, $\phi$ might be measured in radians, yet $||a_t-a_{t-1}||$ is not at all in radians. Every physicist will tell you that this means something is wrong!

Therefore, my question is - How can my reward depend on drastically different phenomenons that do not relate to each other?

Answer 1

First thing I'd say is that I've found Cross Validated to not be the best place for questions about reinforcement learning (at least on such depth). I'd highly recommend ai.stackexchange and/or r/reinforcementlearning on Reddit.

Now, as to your question. I understand where you're coming from, that there isn't some straightforward, unit-less way to combine these metrics. However, you have to remember that RL rewards don't need to be based in the physical, logical laws of the universe. Your goal, when designing a reward function, is very practical/empirical - design a function that makes the agent do what you want it to do. The only thing you might want to be aware of if you make a reward func too divorced from physical reality is that there might be things that the reward function "accidentally" allows while being catastrophic in the real setting.

All that said, here are some thoughts/ideas:

1. You have what seems to be a pretty typical set up. You might find this video helpful: https://www.youtube.com/watch?v=bD6V3rcr_54&t=364s
2. You might consider simply giving a flat reward for every time step that the agent's phi is in the correct range, and no reward otherwise (or a negative reward). As to incorporating the torque constraint, you might again consider just giving a negative reward. As to the magnitude of the rewards, you might have to just use trial and error. Make phi-based reward +1 per second and torque-based punishment -5 every time. Observe the behavior. If you find that the agent still makes big torque adjustments, it must mean that the agent thinks that it's STILL worth doing those. Make the punishment bigger. If agent isn't staying in the range, make phi reward bigger per second and add small punishment for all other zones etc. My understanding is that the approach to these things is really very empirical (don't need to fret too much about how the units translate to real life as long as your problem is reasonably represented).
3. Look into action masking. If it were me, I think "teaching" the agent that big torque changes are bad would be a lot harder than making such movements impossible. Look into action masking for continuous action spaces. I'm only familiar with masking for discrete action spaces, but I'm sure there's a continuous solution. The idea here is just to make it impossible for your agent to take a drastic action - this simplifies your problem, as your agent now just has to worry about staying in the phi bounds using the tools it has (small-medium adjustments).