passage-eval API reference¶

Per-sample cost regret. Returns NaN where the oracle cost is non-positive.

Parameters¶

predicted_costs : array_like, shape (N,) Cost of the decoded predicted path under the cost model. Use np.nan for samples where the decoder failed to produce a path. oracle_costs : array_like, shape (N,) Cost of the exact A* oracle path under the same cost model. eps : float Numerical floor for the denominator.

Return median / p90 / p95 / mean / geomean of a regret distribution.

The geometric mean is computed on 1 + regret_i and reported as exp(mean(log(1 + r))) - 1; clipped at -1 + 1e-12 to keep the log finite when an unusually-decoded path is sub-zero in regret (which can happen at numerically degenerate corners of the ODD).

Per-sample normalised ASSD; nan for empty paths.

Returned array has shape (N,) and units of pixels per unit oracle path length. Aggregate with np.nanmean / np.nanmedian over a batch.

Per-sample symmetric percentile-Hausdorff in pixels.

Per-sample raw path-mask IoU on a fixed grid.

Diagnostic only — drift between path_iou and the cost-regret family is what this metric is reported for, not as a promotion signal.

Return connection, obstacle violation and traversal count for one path.

Invalid coordinates are never rounded or clipped into a valid grid cell. An obstacle crossing is reported separately from start-to-goal connection.

Fraction of samples whose decoded path connects start to goal.

A path is "successful" iff: (i) it has at least two cells, (ii) consecutive cells are 8-connected, and (iii) when starts and goals are provided, the first / last cells match the prescribed endpoints.

Returns NaN for an empty batch.

Fraction of samples whose decoded path crosses any obstacle pixel.

Each obstacles[i] is an (H, W) boolean array (1 = forbidden). Returns NaN for an empty batch.

Mean obstacle-cell traversal count, conditioned on a violation.

Distinguishes 'clipped a corner' (1) from 'drove through the building' (50). Returns NaN if the batch contains no violation.

Validated per-sample evidence; censored costs retain failed decodes.

Aggregate validated per-sample arrays without rerunning inference.

Six metrics; costs include censored failures, geometry uses valid paths.

Costs passed to this lower-level API must already use the analytic scene model. Use score_planner to recompute them directly from candidate paths.

Paired-bootstrap CI on the difference of a metric across paired samples.

Both arrays are indexed identically (sample-paired), so the sample-difficulty variance cancels under the resample. metric_fn is applied to each resampled pair and the difference recorded.

Return predicted costs with a \(K \gg c^{\star}\) penalty on failed decodes.

The paper defines a censored penalty \(c^{\text{pred}}_i = K\) "on a decode that fails to connect \(s\) to \(g\) or that crosses an obstacle". The canonical PASSAGE penalty value is K = k_factor * oracle_cost with k_factor = 1e6.

Parameters¶

predicted_costs, oracle_costs : array_like Per-sample costs in matching order. Pass np.nan (or the already-censored sentinel) for samples whose decode failed - this function will rewrite those entries. decoder_succeeded : array_like of bool, optional True iff the canonical decoder produced an \((s\to g)\) path. Mutually consistent with success_rate. obstacle_violated : array_like of bool, optional True iff the decoded path crosses an obstacle pixel. k_factor : float Multiplier on the oracle cost used as the censored penalty.

Returns¶

np.ndarray of shape (N,) Predicted costs with failed decodes rewritten to k_factor * oracle_cost. Entries with non-finite oracle cost remain non-finite.

Paired bootstrap on T(A) - T(B), preserving draw order.

Pairs are anchored on the same sample index: the pairing cancels sample-difficulty variance because both methods are evaluated on the same indices. With blocks, each draw samples the observed number of whole geographic blocks with replacement, retaining task weights. Without blocks this lower-level helper uses independent task resampling.

For cost_regret_median and friends pass statistic=np.median. The mean default agrees with the historical mean of paired differences. Nonlinear reductions intentionally use the difference of aggregates.

Parameters¶

per_sample_a, per_sample_b : array_like, shape (N,) Per-sample metric values, paired by index. statistic : callable Reduction applied separately to each method, including inside draws. n_bootstrap, alpha, seed Bootstrap parameters; alpha = 0.05 is the paper's default. return_draws : bool If True also return the (n_bootstrap,) draw vector for descriptive inspection. Ordinary bootstrap tails alone do not define the valid paired null test required for confirmatory Holm correction. nan_policy : {"raise", "omit"} Missing values raise by default. Explicit omission draws the complete roster first, then reduces each method's finite subset separately. An empty subset in any draw makes its CI unavailable; draws are retained.

Average seed contrasts within each shared task/block bootstrap draw.

The result contains B = n_bootstrap_per_seed draws of the seed mean, and per-seed estimates/CIs in per_seed. seed controls geographic resampling; seeds identifies fitted training seeds. This intentionally replaces the historical seed-union mixture of S * B draws.

Inputs are mappings keyed by exactly seeds, or sequences in that order. All arrays must represent the identical ordered task roster. Use compare_primary_metrics to validate explicit task identities and the complete preregistered seed/metric family.

Holm–Bonferroni step-down correction on a family of two-sided p-values.

Implements the standard step-down procedure: order p-values ascending, and reject at step \(k\) iff \(p_{(k)} \leq \alpha / (m - k + 1)\) and every earlier hypothesis was rejected. Returns a per-metric record carrying the original p-value, the Holm-adjusted p-value (\(\min(1, (m - k + 1) p_{(k)})\) with monotonised cumulative max), and the reject decision at family-wise alpha.

The order of the input mapping is ignored; results are returned in the original metric order (Python 3.11+ dicts preserve insertion). A NaN represents an unavailable test: it remains NaN and cannot reject, while still counting in the declared family size. Other values outside [0, 1], including infinities, raise instead of becoming evidence.

Reduce per-metric paired-bootstrap deltas + Holm flags to a verdict.

Pareto-dominant: every metric votes the same direction. Indistinguishable: no metric votes either way. Trade-off: demonstrated wins in both directions. Partial evidence: demonstrated wins in one direction with inconclusive remaining metrics. This generic helper judges only the supplied family; use compare_primary_metrics for all six preregistered primary metrics.

Compare all primary metrics on an explicit, frozen three-seed roster.

Each seed supplies task_id, regret, assd, h95, success, obstacle_violation and valid_path arrays in identical task order. Regret must already retain censored failures; reliability uses all tasks. Geometry reduces each method's own successful, obstacle-free subset. The caller is responsible for matching data, operating conditions, work limits and frozen checkpoints; this function never loads or evaluates one.

Geographic blocks are sampled identically for both methods, every seed and every metric. Within each draw, compute each seed's prescribed aggregate contrast and then average the three contrasts. Per-seed CIs and method-specific denominators accompany the seed-mean interval. A geometry draw with an empty subset makes the CI unavailable; no draw is discarded.

Holm uses only the six explicitly supplied p-values from a predeclared paired test of this same block/seed-mean contrast. The test name and its assumptions are required metadata, not automatic evidence of validity. The legacy bootstrap-tail proxy is never used. Without a declared test, estimates/CIs are available but no confirmatory win is returned. NaN tests remain unavailable and retain their place in the six-test family. Claims across more method pairs require a larger prespecified correction family.

F-score of boundary agreement at a tolerance of tolerance pixels.

Standard segmentation-challenge boundary-F at tolerance d is precision = |pred ∩ dilate(gt, d)| / |pred| recall = |gt ∩ dilate(pred, d)| / |gt| F = 2 PR / (P + R) Returns NaN when both masks are empty.

Number of 8-connected components of mask > 0.

Do start and goal cells share an 8-connected component of mask>0?

Mean per-cell Bellman residual on reachable non-goal cells, normalised.

For each reachable non-goal cell s, residual(s) = | V_hat(s) - min_n ( c(s, n) + V_hat(n) ) | Normalised by mean(|V_hat|) over the reachable subset.

Fraction of cells where the local 8-argmin of the predicted field matches the oracle. Restricted to reachable, non-goal, non-obstacle cells.

A regression model with descent_direction_agreement = 0.99 produces oracle-equivalent paths under a greedy decoder regardless of MAE.

Scale-and-shift-invariant MAE.

Solves for a, b = argmin || a * pred + b - oracle ||_1 via a closed-form least-squares fit, then reports the L1 residual on the fitted scale. Cross-cost-model comparable; invariant to the per-sample V normalisation.

Fraction of neighbour pairs with non-trivial oracle margin where the predicted field ranks them the same as the oracle.

A scale-invariant ordering check that complements MAE: a field can have a low MAE and still rank neighbouring cells in the wrong order, which misleads a descent decoder.

Coefficient of variation \(\sigma / \mu\) of the per-seed cost_regret_p90 (axis "Stab").

A planner whose three-seed CR-p90 has a CV above cv_threshold is flagged as training-unstable. The default cv_threshold is 0.25.

Difference in the mean of a primary metric between the geographic hold-out split and a matched-marginal random split (axis "Geo").

Both inputs are per-sample arrays for the same metric. The output value is mean(holdout) - mean(matched_random); higher_is_better sets which sign counts as worse.

The axis fires when the hold-out mean is worse than the matched random mean. The flag uses the sign of the difference only; it is not a significance test.

Difference between the top-quantile (boundary) slice and the full-split mean of a primary metric (axis "ODDe").

stratifier_per_sample carries one of the four declared per-sample ODD axes (obstacle coverage, super-ellipse exponent, elevation band, start-goal distance). Samples whose stratifier sits in the top top_quantile are designated the "boundary" slice. The axis value is \(\Delta = \mathrm{mean}(\text{boundary}) - \mathrm{mean}(\text{full})\).

For smaller-is-better metrics, a positive \(\Delta\) means the planner worsens at the envelope edge — the axis fires.

Same-device latency ratio between planner and A\(^{\star}\) oracle (axis "Eff").

The axis value is the speed-up \(\mathrm{speedup} = T_{\text{oracle}} / T_{\text{planner}}\).

A speed-up below speedup_threshold = 1.0 means the planner is slower than the oracle that produced its labels — the axis fires.

Inputs may be scalars or arrays; arrays are summarised by their median.

expansions(decoded planner on V_hat) / expansions(plain A* on true cost).

Reported as median, p90 and geometric mean. < 1 means the surrogate accelerates A*; ≈ 1 means no acceleration; > 1 means the surrogate adds search effort.

Fraction of samples whose canonical decoder produced a path without triggering the A*-recovery fallback.

This legacy measure excludes only the explicitly reported fallback stage. It does not receive a bounded-repair flag and therefore cannot establish that a successful decoder ran without repair. Use the scored direct_success_rate and repair_rate to distinguish that case.

Time the canonical decoder over a fixed batch of model outputs.

The decoder is run once per element of inputs. Latency is the mean per-sample wall-clock, and the percentile breakdown is computed over the per-sample times.

Forward + decoder per sample, returning percentile latency.

Each inputs[i] is fed to model and the output to decoder_fn. End-to-end time per sample = forward + decode. Use this to populate the Lat. (ms) column of the headline table. Select a PyTorch device explicitly to benchmark an accelerator.

Benchmark the bare forward pass of model on synthetic input.

Parameters¶

model A torch nn.Module-like callable (or a wrapper exposing network). Plain Python callables also work — they will be treated as bare functions on numpy input when torch is not present. input_shape Shape of one synthetic input batch (e.g. (1, 3, 256, 256)). device PyTorch device. Defaults to CPU. Select CUDA explicitly to use it. dtype "fp32" (default), "fp16" or "bf16".

Deterministic SHA-256 over a calibration-decisions mapping.

Renders the mapping as canonical JSON (sort_keys=True, separators=(",", ":")) before hashing so two manifests with the same logical content always produce the same hash. Path, set, and tuple values are normalised for serialisation.