Overview of PVF. Data collection: trajectories are collected by running a nominal policy $\pi(x)$ on black-box systems (e.g., legged robots, manipulators, quadrotors) subject to stochastic disturbances $w$, and stored as transition pairs $(x, x')$ in a replay buffer. Self-supervision: piecewise target labels $V_\text{target}(x)$ are assigned based on constraint violation ($0$), task completion ($1$), or bootstrapped value evaluation ($V_\theta(x')$). Training: a neural network $V_\theta(x)$ is optimized via a temporal-difference loss to output the continuous reach-avoid success probability.
Abstract
Reach-avoid sets, defining the states from which a task can be completed while avoiding unsafe regions, are fundamental tools for deploying robots in environments subject to disturbances and uncertainty, as well as for runtime task-feasibility monitoring, allowing prediction of whether a state is safe and will lead to successful task completion. Classical Hamilton-Jacobi Reachability (HJR) and its recent neural extensions (e.g., Lipschitz Continuous Reachability Learning, DeepReach) treat disturbances adversarially, computing the reach-avoid set under worst-case disturbances from a bounded set. This guarantees safety but can yield overly conservative sets when disturbances are stochastic rather than adversarial, and additionally requires the disturbance set to be known or specified in advance.
We propose Probabilistic reach-avoid Value Function (PVF), a self-supervised method, applicable to black-box systems, for computing reach-avoid sets as the states with a certain reach-avoid probability under stochastic, potentially unbounded disturbances. We evaluate our approach against HJR-based baselines on systems of increasing complexity, up to a Hopper locomotion task, and show that our probabilistic sets are less conservative and more accurate than adversarial HJR sets when disturbances are non-adversarial.
Key Contributions
Self-supervised probabilistic reach-avoid sets
A TD-learning method that computes, for each state, the probability of reaching a target set without ever entering a constraint set, under stochastic disturbances with potentially unbounded support. It treats the system as a black box, with no model, gradients or disturbance distribution required.
Interpretable, less conservative sets
The learned value is an actual success probability rather than a worst-case margin. Under non-adversarial disturbances, the resulting reach-avoid sets are more interpretable, less conservative and more accurate than those of adversarial HJR methods.
Method
Given a black-box stochastic system and a fixed policy, PVF learns, for every state, the probability of reaching a target set before entering a constraint set.
Only transitions needed
The method only needs to sample transitions from a simulator or real-world rollouts. Target and constraint sets are specified through indicator functions $\mathcal{I}_\mathcal{T}$ and $\mathcal{I}_\mathcal{C}$, not level-set equations.
Undiscounted recursion
The event of reaching the target before the constraint set decomposes recursively over one step, giving an undiscounted Bellman equation for the hitting probability with boundary values $1$ on $\mathcal{T}$ and $0$ on $\mathcal{C}$.
Self-supervised targets
The unknown expectation over the dynamics is replaced by the current estimate at the observed next state, and a neural network $V_\theta$ is trained by TD regression with a Polyak-averaged target network.
Setting. We consider stochastic discrete-time systems $x' = f(x, u, w)$, with state $x \in \mathcal{X} \subseteq \mathbb{R}^n$, control $u \in \mathcal{U} \subseteq \mathbb{R}^m$ and disturbance $w \sim p_\mathcal{W}$ drawn from a possibly unbounded distribution. The transition function $f$ is a black box: it can only be queried through a simulator or real rollouts. Given a fixed policy $\pi$, a target set $\mathcal{T}$ and a constraint set $\mathcal{C}$, we want the reach-avoid probability
$$V(x) \triangleq \mathbb{P}\big(\tau_\mathcal{T} < \tau_\mathcal{C} \mid X_0 = x\big),$$
where $\tau_\mathcal{T}$ and $\tau_\mathcal{C}$ are the first hitting times of the two sets. The sets need not be given in closed form: it is enough to have indicator functions $\mathcal{I}_\mathcal{T}(x)$ and $\mathcal{I}_\mathcal{C}(x)$, equal to $1$ when $x$ is in the target or constraint set and $0$ otherwise, with $\overline{\mathcal{I}} = 1 - \mathcal{I}$ denoting their negation.
Recursion. At each step $i$, the state $x_i$ is either in the target set (event $T_i$), in the constraint set ($C_i$), or in neither ($N_i$); these events are mutually exclusive. Let $S_0$ be the event of eventually reaching the target before violating any constraint, starting from step $0$. Either the target is reached immediately, or the state is in neither set and success happens from the next step onward: $S_0 = T_0 \lor (N_0 \land S_1)$. Translating this into probabilities and conditioning on the observed state gives the reach-avoid value function:
$$V(x_0) = \mathcal{I}_{\mathcal{T}}(x_0) + \overline{\mathcal{I}_{\mathcal{C}}}(x_0)\, \overline{\mathcal{I}_{\mathcal{T}}}(x_0)\, \mathbb{E}\big[V(x_1) \mid X_0 = x_0\big]$$
Since the transition distribution is unknown, the expectation is approximated by sampling transitions and applying temporal-difference learning with the piecewise target
$$V_\text{target}(x, x') = \begin{cases} 0 & \text{if } x \in \mathcal{C} \\ 1 & \text{if } x \in \mathcal{T} \\ V(x') & \text{otherwise} \end{cases} \qquad \min_\theta \sum_{(x,x') \in \mathcal{P}} \big\| V_\theta(x) - V_\text{target}(x, x') \big\|^2$$
Assuming the system eventually reaches $\mathcal{T} \cup \mathcal{C}$ almost surely, the two boundary conditions uniquely determine the reach-avoid probability and supply terminal supervision for TD learning. Although data is collected with a finite episode length, the bootstrapped targets make the effective horizon grow with the number of training epochs. The probabilistic reach-avoid set at level $\ell$ is then the superlevel set $\{x : V_\theta(x) \geq \ell\}$.
Probabilistic vs. Adversarial Sets
On a scalar toy problem from the LCRL paper, $x' = 1.01\,x + 0.01\,(u + w)$ with $|u| \le 1$ and Gaussian noise truncated at $|w| \le 4$, the worst-case disturbance prevents the system from ever reaching the target, while the noise is in fact not adversarial.
Reach-avoid value functions on the toy problem. PVF ($V_\text{PVF}$, left) vs. the adversarial LCRL baseline ($V_\text{LCRL}$, right). Green regions indicate states identified as safe, i.e. able to reach the target set without violating constraints.
With this level of noise, the HJR solution assumes the disturbance always takes its worst-case value within the bound. Under that adversary the system can never be steered into the target from outside it, so the HJR reach-avoid set collapses to the target set itself. PVF instead samples the disturbance from its actual distribution, where such worst-case sequences are extremely unlikely. It therefore identifies a much larger set of states from which the target is reached with high probability. When noise is random rather than adversarial, the probabilistic set is substantially less conservative.
Results
We compare PVF against LCRL on Drones Racing (12-dimensional state) and Hopper (12-dimensional state, non-smooth contact dynamics), using the same policy for all methods. Ground truth is obtained from 4000 Monte Carlo rollouts per state, certified with Wilson score intervals at 99.5% confidence and safety level $\ell = 0.99$.
Drones racing
An ego drone must safely overtake an opponent drone and reach a gate first. The opponent's acceleration is uncertain, modeled as a zero-mean Gaussian truncated at $3\sigma$ with norm bound $2~\text{m/s}^2$. At the nominal thresholds ($0.99$ for PVF and $0$ for LCRL), PVF yields remarkably fewer false negatives (missed alarms, the most dangerous failure) and moderately fewer false positives. LCRL only matches PVF's false negatives when its threshold is raised above $0.1$, at the cost of over 6% false alarms.
Classification against certified ground truth, Drones Racing. PVF (left) and LCRL (right) over the initial horizontal position $(x, y)$ of the ego drone, with the opponent starting at $x = 0.4$ m, $y = 0$ m. LCRL shows notably more false negatives, particularly for $y > -1.5$ m.
Hopper
The Hopper starts at a random state with horizontal position $x_h \in [-10, 60]$ m and must reach $x_h \ge 50$ m without falling, under Gaussian velocity disturbances truncated at $4\sigma = 0.15$ m/s. PVF is again more accurate on average in both false negatives and false positives at the nominal thresholds: the median false-negative rate of LCRL is about 4% higher, and PVF achieves nearly 0% false positives.
Drones Racing.
Hopper.
Accuracy comparison across classification thresholds. False-negative vs. false-positive rates of PVF and LCRL over three seeds (median, min and max), with the classification threshold reported next to each point.
Why the Target Set Matters
Even when the main goal is avoiding failures, the target set is essential. We train an avoid-only variant on Hopper, equivalent to SQRL without discounting, where the TD target is $0$ in $\mathcal{C}$ and $V(x')$ otherwise.
| Metric (avoid-only, Hopper) | Value (%) |
|---|---|
| True positive | 74.6 |
| False positive | 23.6 |
| True negative | 0.0 |
| False negative | 0.0 |
| Unclear cases | 1.8 |
Avoid-only ablation. Without a target set, the network classifies every state as unsafe and never detects safe states: since training is self-supervised, the only grounding signal comes from failures, which pulls all estimates towards them.
Application: Model Predictive Shielding
The learned PVF and its backup policy are used as a safety filter for an aggressive nominal Hopper policy. The shield switches to the backup policy whenever the predicted success probability at the next state falls below $0.99$.
| Noise (4σ) | 0.10 | 0.15 | 0.20 |
|---|---|---|---|
| Shielded | 99.1 ± 1.6 | 97.2 ± 4.2 | 91.0 ± 8.3 |
| Unshielded | 97.1 ± 5.2 | 90.8 ± 7.7 | 83.1 ± 10.0 |
Shielding results. Success counts out of 100 trials (mean ± std across 11 seeds), with identical disturbance realizations for the shielded and unshielded runs. Shielding consistently reduces failures, even at noise levels beyond the $4\sigma = 0.15$ used to train the PVF, with the gap widening to 7.9% at $4\sigma = 0.20$.
Shielded vs. unshielded rollouts under identical disturbance realizations. In each video, the nominal policy with PVF shielding (left) reaches the target, while the nominal policy alone (right) falls.
Training noise level ($4\sigma = 0.15$ m/s) — example 1
Training noise level ($4\sigma = 0.15$ m/s) — example 2
Higher noise ($4\sigma = 0.20$ m/s) — example 1
Higher noise ($4\sigma = 0.20$ m/s) — example 2
BibTeX
@article{fontanari2026pvf,
title = {{Temporal Difference Learning of Probabilistic Reach-Avoid Value Functions}},
author = {Fontanari, Elias and Crestaz, Pietro Noah and Saveriano, Matteo and Del Prete, Andrea},
journal = {arXiv preprint arXiv:2610.28106},
url = {https://arxiv.org/abs/2610.28106},
year = {2026},
}