SeCBAD: An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Context

Problem Setting

The latent context / MDP can change abruptly at an unknown time inside the same episode.

This is different from methods such as DARLING, where the MDP is assumed to remain fixed within one episode.

SeCBAD assumes that the context is piecewise stable:

$$A,A,A,A\to B,B,B\to C,C,C,C,\dots$$

The regime stays unchanged for some stochastic amount of time and then changes abruptly.

The method tries to answer two hidden questions:

1. Where did the current regime begin?
2. What is the current regime?

Equivalently, SeCBAD jointly infers:

$$p(c_t,G_t\mid \text{history})$$

where:

• $c_t$: current latent context / regime
• $G_t$: how long the current regime has lasted so far

The segment structure itself is latent: the agent is not told where one regime ends and the next begins.

Once SeCBAD estimates where the current segment began, it can use only the data since the estimated latest change point to infer the current context. This avoids contaminating current-regime inference with data generated by previous regimes.

Context Inference and Context Belief

The true context $c_t$ is hidden.

The agent therefore maintains a probability distribution over possible latent contexts:

$$p(c_t\mid \text{history})$$

This is called the context belief.

• context inference: the process of estimating the hidden context from data
• context belief: the probability distribution over possible contexts
• inferred context belief: the context belief produced by the inference model

The history is the trajectory observed by the agent, including states, actions, rewards and transitions.

After the context belief is inferred, SeCBAD augments the ordinary RL state:

$$s_t\to (s_t,b_t)$$

where $b_t$ is the inferred context belief.

The policy therefore becomes:

$$\pi (a_t\mid s_t,b_t)$$

The policy does not receive the true regime ID. It receives the inferred belief about the current regime.

Latent Situational MDP

The paper formalizes the changing environment as a Latent Situational Markov Decision Process (LS-MDP):

$$M=(S,A,C,X,G,T,R,\gamma )$$

where:

• $S$: state space
• $A$: action space
• $C$: latent context space
• $X$: observable context-related information used as auxiliary information during training
• $G$: segment-duration variable
• $T$: transition function
• $R$: reward function
• $\gamma$: discount factor

The current context changes both the transition and reward functions:

$$T:p(s_{t+1}\mid s_t,a_t,c^h)$$ $$R:p(r_t\mid s_t,a_t,c^h)$$

Therefore a context change corresponds to an actual MDP change.

Segment notation

$h$ is the segment index.

At the beginning of segment $h$, the environment samples:

$$G^h\sim p_G(G)$$ $$c^h\sim p_c(c)$$

where:

• $G^h$: total duration that segment $h$ will eventually have
• $c^h$: context / regime active during segment $h$

During that segment, the context remains fixed:

$$c_t=c^h$$

$G_t$ is different from $G^h$.

• $G^h$: total length of segment $h$
• $G_t$: how long the current segment has lasted so far at time $t$

For example, if a segment will eventually last five steps:

$$G^h=5$$

then during the segment:

$$G_t=1,2,3,4,5$$

and when a new segment starts, $G_t$ resets to 1.

Observable Context $X$

During training, the paper assumes access to an auxiliary noisy observation:

$$x_t\sim p(x_t\mid c_t)$$

$x_t$ is not the RL transition tuple.

It is an additional context-related signal correlated with the hidden context.

For example, the hidden context may describe terrain properties, while $x_t$ could be some noisy simulator-provided measurement related to those properties.

For my purpose, $X$ can be viewed as optional privileged training information.

The more important setting for my work is to infer context directly from the ordinary streaming RL sample:

$$(s_t,a_t,r_t,s_{t+1})$$

without requiring privileged context labels or measurements.

Joint Inference

SeCBAD decomposes joint context / segment inference as:

$$p(c_t,G_t\mid {\tau }_{1:t})=p(c^{t-G_t+1:t}\mid G_t,{\tau }_{t-G_t:t})p(G_t\mid {\tau }_{1:t})$$

where:

$${\tau }_{1:t}=(s_1,a_1,r_1,s_2,\dots ,a_{t-1},r_{t-1},s_t)$$

is everything observed up to time $t$.

The equation can be understood in plain language as:

First estimate where the current segment started. Then, using only data from that segment, estimate what the current context is.

Or more compactly:

$$p(C,G\mid \text{history})=p(C\mid G,\text{current-segment data})p(G\mid \text{history})$$

This is just the ordinary probability product rule:

$$p(A,B)=p(A\mid B)p(B)$$

The term posterior means a probability distribution after observing data.

For example:

• prior: belief about segment length before seeing the trajectory
• posterior: updated belief after observing transitions and rewards

Context Encoder

Section 3.1 first considers the easier case where the current segment length $G_t$ is known.

Then the problem becomes:

Given the transitions belonging to the current segment, what latent context best explains them?

SeCBAD uses an inference network / encoder $q_{\varphi }$.

Conceptually:

$$\text{current-segment trajectory}\to q_{\varphi }\to \text{latent context distribution}$$

The encoder is implemented using a GRU.

It does not output a discrete regime label such as "A" or "B". Instead, it outputs a distribution in a continuous latent context space.

The latent context distribution is modeled as a diagonal Gaussian whose mean and variance are produced by the encoder.

Encoder-Decoder Training

The encoder and decoder are trained before deployment.

The encoder tries to infer a useful latent context representation.

The decoder then asks whether that inferred context can explain what happened in the current trajectory segment.

The decoder contains models for:

• next-state transition
• reward
• optional observable context $x_t$

Conceptually:

$$(s_t,a_t,c)\to \text{predict }{s}_{t+1},r_t$$

and, during training, optionally:

$$c\to x_t$$

The context representation is therefore trained to contain information useful for explaining how the environment currently behaves.

Equation 2

Equation 2 is essentially a VAE-style objective.

It contains two ideas:

1. The inferred context should reconstruct / explain the observed segment well.
2. The latent context distribution should remain close to a reasonable prior.

In plain language:

Learn a context representation that is informative enough to explain the current environment, but regularized enough that it does not simply memorize arbitrary trajectories.

The KL-divergence term is the regularization term.

The full ELBO derivation is not important for my project.

Equation 3

Equation 3 simply expands the reconstruction term step by step.

For every transition in the current segment, the decoder asks:

Given the previous state, action and inferred context, how likely are the observed next state and reward?

These likelihoods are summed across the current segment.

The important design idea is that only data believed to belong to the current segment are used.

If the true history is:

$$AAAAA\mid BBB$$

the encoder should infer the current $B$ context from:

$$BBB$$

rather than:

$$AAAAABBB$$

This is the main benefit of estimating the segment boundary.

Belief Context as a Mixture Distribution

The true $G_t$ is unknown.

Therefore SeCBAD considers multiple possible current segment lengths.

For example:

$$p(G_t=1)=0.1$$ $$p(G_t=2)=0.2$$ $$p(G_t=3)=0.6$$ $$p(G_t=4)=0.1$$

Each possible segment length produces a different context belief because the encoder sees a different trajectory window.

Conceptually:

• assume $G_t=1$ -> context distribution 1
• assume $G_t=2$ -> context distribution 2
• assume $G_t=3$ -> context distribution 3
• ...

The final context belief combines all of these distributions, weighted by the posterior probability of the corresponding segment length.

This is why it is called a mixture distribution.

In plain language:

For every plausible point where the current regime may have started, infer a context under that assumption, and combine those context beliefs according to how plausible each segment length is.

For implementation simplicity, the paper also considers selecting the most likely segment length $G_t^{\ast }$ and using only the corresponding context belief. They report little performance loss from this approximation.

Policy Optimization

Once the context belief $b_t$ is inferred, the RL part is relatively simple.

Instead of giving the policy only:

$$s_t$$

SeCBAD gives:

$$(s_t,b_t)$$

Therefore:

$$\pi (a_t\mid s_t)$$

becomes:

$$\pi (a_t\mid s_t,b_t)$$

The same belief-augmented state idea is used for the value / reward / transition formulation under the belief-MDP interpretation.

The belief is not an additional reward.

It is additional information included in the state representation.

SeCBAD uses PPO for policy optimization.

Training vs Deployment

During training:

• the context encoder $q_{\varphi }$ is trained
• the decoder $p_{\theta }$ is trained
• the PPO policy is trained
• auxiliary observable context $x_t$ may be available

During deployment:

• the encoder is frozen
• the decoder is frozen
• the policy is frozen
• $x_t$ is no longer available

The system then uses observed transitions and rewards to estimate:

1. the current segment-length posterior
2. the current latent-context belief

and feeds the inferred context belief to the policy.

Therefore SeCBAD is mainly:

$$\text{pretrain context inference}\to \text{freeze it}\to \text{infer context online during deployment}$$

It is not a lifelong learner that continues updating its context encoder and policy indefinitely during deployment.

Relevance to My Research

The most useful idea from SeCBAD is the decomposition:

$$\text{detect / infer regime boundary}\to \text{infer current latent context}\to \text{condition policy on context}$$

In particular, it provides a concrete example of learning:

$$\text{trajectory}\to z_t$$

where $z_t$ is a compact latent representation of the current regime.

Another useful idea is that stale data from previous regimes should not contaminate inference of the current regime.

However, SeCBAD itself does not satisfy the strict streaming setting I am interested in.

Major mismatches:

• requires a separate training phase
• context encoder / decoder are pretrained and then frozen
• uses trajectory segments / past history
• evaluates multiple possible segment lengths
• relies on a history window rather than only the newest transition
• uses PPO rather than a strict streaming actor-critic
• does not continue learning indefinitely during deployment

Therefore I should not directly implement SeCBAD on top of AVG.

The useful transformation for my setting would be to replace:

$${\tau }_{t-G_t:t}\to c_t$$

with a fixed-size online recurrent context state:

$$z_{t+1}=f_{\psi }(z_t,s_t,a_t,r_t,s_{t+1})$$

The recurrent state $z_t$ summarizes relevant history without storing the trajectory itself.

Memory then remains fixed rather than growing with segment length.

The policy could become:

$$\pi (a_t\mid s_t,z_t)$$

and similarly:

$$Q(s_t,a_t,z_t)$$

This suggests the research question:

Can SeCBAD-style latent regime inference be achieved in a truly streaming learner using only the current transition and a fixed-size recurrent context state, without replay, stored trajectory segments, pretrained context models, or per-regime policies?

Main Takeaway

SeCBAD's main contribution is not simply allowing an MDP to change inside an episode.

Its important idea is:

Explicitly infer both where the current piecewise-stable regime began and what latent context is currently active, then condition the RL policy on the resulting context belief.

For my project, the most valuable component is the latent regime representation:

$$\boxed{\text{experience}\to \text{compact context representation}\to \text{context-conditioned policy}}$$

The missing piece is how to make this mechanism truly streaming and lifelong under bounded memory.