Q-Shaped Options for Hierarchical Reinforcement Learning

Learning to tackle long-horizon, goal-conditioned tasks requires an agent to reason over extended timescales and act across a broad range of states.

In goal-conditioned two-tiered Hierarchical Reinforcement Learning (HRL) a high-level policy outputs a high-level action or option conditioned on the current state and goal; a low-level policy then conditions on this option as well as the current state to output a primitive action. In maze navigation, for example, the option might be to reach ''a point two metres down'', while the primitive action could be applying torques to an individual joint.

In principle, HRL addresses both challenges of long-horizon, goal-conditioned tasks through the interaction between action (temporal) and state (spatial) abstraction.

Firstly, by using an action abstraction to represent temporally extended behaviour as options, a hierarchical policy reduces the effective decision horizon. The high-level policy reasons over options instead of primitive actions, while the low-level policy reasons to pursue the option rather than the distant goal.

Secondly, by enabling different state abstractions at each level of the hierarchy, a hierarchical policy can achieve coarser state abstraction than a non-hierarchical policy if each level discards different information irrelevant to its part of the decision process. Intuitively, this permits greater data aggregation for learning. For example, the high-level can aggregate all examples at a certain coordinate irrespective of pose, which is too fine-grained for high-level control, while the low-level can aggregate all examples with a certain pose irrespective of coordinates, since reaching ''a point two metres down'' requires the same actions wherever the agent is.

The problem is that realising these two benefits of a hierarchical policy depends on learning an appropriate action abstraction.

To improve stability and avoid collapsing to degenerate solutions (where options terminate at every step, or run to the end of the episode), decoupled approaches train each policy separately using fixed-length trajectory segments. The action abstraction maps these segments to the option space, serving as actions for the high-level policy and goals for the low-level policy during training.

The issue is that current HRL algorithms learn abstractions that fail in one of two ways.

Some discard distinctions between options that are needed for optimal control. For example, HIQL learns the action abstraction through its single action-free state-value function, but value equivalence does not imply action equivalence. This limits the quality of the recoverable policy and undermines hierarchy altogether.

Others retain distinctions between options that are irrelevant for control. For example, SHARSA and DSHARSA use Absolute Subgoal Options (i.e. they bypass learning an action abstraction by simply specifying the entire state that the low-level policy should reach). Similarly, OPAL uses a surrogate objective to learn an option space using a $\beta$-VAE trained to reconstruct the entire action sequence, meaning that the option encodes how a segment was executed, instead of its outcome. Both of these ''action abstractions'' preserve horizon reduction but forfeit coarser state abstraction, as functionally equivalent behaviours remain distinct outputs of the high-level policy and distinct inputs to the low-level one.

We characterise three desiderata for an action abstraction to enable both horizon reduction and data aggregation. We also introduce QSO, to address all three. QSO builds on an architecture with separate state-value functions, Q functions and policies at each level of the hierarchy.

To address our first desideratum, we encode only the current and target states $(s_t, s_{t+k})$ to represent the outcome of a $k$-step trajectory segment $\tau_{t:t+k}$ rather than the exact state-action sequence that produced it.

To address our second desideratum, we shape the abstraction jointly with the low-level Q function: $Q_l(s_t, \phi_\omega(s_t, s_{t+k}), a_t)$. Intuitively, if two admissible target states require different actions from the same current state, the low-level Q function must represent different sets of action values. Minimising the Bellman error therefore pressures the action abstraction to map the current and target state pairs to different points in the option space. In our paper, we introduce a formal definition of a control-sufficient action abstraction and prove that low-level Q shaping satisfies control-sufficiency (note that this is actually stronger than control sufficiency).

To address our third desideratum, we additionally shape the option space jointly with the high-level Q function: $Q_h(s_t, g, \phi_\omega(s_t, s_{t+k}))$. Functionally equivalent state-target pairs induce similar high-level action values, so distinguishing between them provides little predictive benefit to $Q_h$. Note that, unlike control sufficiency, this invariance is not explicitly enforced by the objective, instead simply providing an inductive bias towards it under shared function approximation and gradient-based optimisation. Empirically, we find that this is sufficient to form coherent clusters in the option space.

We first evaluate QSO’s downstream performance when learning from a fixed offline dataset.

To assess our second and third desiderata, we introduce ASO, which shares QSO’s policy extraction but, like SHARSA and DSHARSA, uses absolute subgoal options instead of a learned action abstraction. This isolates the effect of QSO’s action abstraction: discarding distinctions necessary for control should yield a worse policy than ASO, whereas discarding only unnecessary ones should improve performance through greater data aggregation.

Table 1 shows that QSO achieves the highest average success rate across the 11 environments, 54% against 49% for the next best method (DSHARSA). Particularly notable is its performance in humanoidmaze, which has the largest action space dimension (21, against at most 8 in the other environments) where it achieves 90% against 82% for the next best method and three times that of DSHARSA. In puzzle-4x4 it achieves 92% against at most 74% for any baseline. In cube-quadruple, the hardest manipulation environment, QSO is the only method with non-zero success on two of the five tasks. QSO performs consistently across environments, achieving at least 65% of the best success rate in all environments, while every other algorithm drops below 35% of the best in at least one.

Is the action abstraction semantically meaningful?

We test whether QSO maps functionally equivalent behaviours to similar parts of the option space by identifying groups of semantically similar behaviours. For each behaviour, we randomly sample trajectory sequences from the validation dataset and compare PCA projections and whitened distances of their action abstraction encodings. An invariant encoder should produce a clear block-diagonal similarity structure. As depicted in Figures 4 and 5, QSO exhibits this structure across both the locomotion and manipulation environments, whereas HIQL also assigns high similarity between behaviour groups, and OPAL and absolute subgoal options produce low similarity within behaviour groups.

How necessary is joint option shaping by both the high and low-level Q functions?

Joint shaping is important for both the learned action abstraction and downstream performance. As shown in Figures 6 and 7, neither Q function alone recovers the clean semantic structure obtained when both are used. We refer to our paper for the downstream RL performances.

Q-Shaping has two main limitations. Firstly, our current method only tackles goal-conditioned tasks with sparse, binary rewards, where the outcome of a trajectory segment is simply the state it reaches. Defining an outcome is much less obvious for other types of data, such as text or images. In other words, how can we map semantically equivalent paragraphs or images to one goal?

Secondly, in stochastic environments, QSO, like all hindsight-based hierarchical methods (including HIQL, OPAL, SHARSA and DSHARSA), is optimistically biased, as it evaluates an option conditioned on the outcome that was actually realised. We need to prevent the high-level policy from repeatedly proposing options that the low-level policy is unlikely to achieve because of the stochastic nature of the environment.

Our current agenda is as follows. Firstly, extending QSO to the online setting, i.e. using the hierarchical policy for more targeted exploration and ''maximally useful'' data collection.

Secondly, addressing the two limitations above. Both are open research questions with several possible approaches, for example progress-adaptive high-level control to reduce the optimistic bias.