Benchmark¶
PASSAGE ships with a self-contained, version-pinned evaluation contract that turns the dataset into a benchmark. The decoded-path metrics, statistical utilities, trustworthiness diagnostics and complete-request workflow live under src/passage-eval. That package is what you hook up to your planner — whether it is a dense-vision model, a graph neural network, a reinforcement-learning policy, or a classical solver — to get a score with identified inputs, procedure, references and metric contract. This section of the documentation is the single entry point for that process. Start with Evaluate your planner to run conform → run → score → compare → report; use the protocol and API pages to interpret the results.
Terms used in the Benchmark pages
- Ordinary input: what a deployed planner sees for one task: elevation in meters, the obstacle mask, the start and goal cells, the graph rules and the budget. It never includes the reference route or its cost.
- Complete request: one timed call of the planner, from the ordinary input to a route. Input preparation, inference or search, decoding, device synchronization, repair and fallback all count. The paper's direct timer excludes independent host validation and scoring; some local runners include legality checks and recosting.
- Qualified reference: a stored reference route that is legal, whose recomputed cost matches the stored cost, and that an exact solve within
reference_max_expansionsconfirms as optimal. A run starts only when the reference of every task is qualified. - Cohort: the frozen list of tasks that a result was computed on (sample IDs, split, objective and obstacle condition). Use the same cohort for paired comparisons.
- Censored cost: the cost given to a route that fails or crosses an obstacle, 10⁶ times the reference cost, so that every failure stays in the regret statistics.
-
Evaluation Protocol
Six primary metrics, contrasts of their prescribed aggregates, shared geographic-block draws across three training seeds, and Holm–Bonferroni correction with declared paired-test evidence. Decisions distinguish Pareto dominance, trade-offs, partial evidence and no demonstrated difference.
-
passage-evalpackage
Install, public API, and module-by-module guide for the evaluation package. It is separate from the dataset generator, so the evaluation rules can be fixed without changing the released data.
-
Paired statistics
Complete six-metric comparison: pairing checks, shared geographic-block draws, three-seed contrasts and declared paired-test evidence. To compare planners with the CLI workflow, use Evaluate your planner.
-
Leaderboard
The entry index for PASSAGE v1.0. Read the paper results separately: they compare Seg, Reg and physical A* on the complete 40,000-scene 256×256 public test split.
-
Evaluate your planner
Wrap ordinary inputs, freeze a specification, qualify references, run the complete procedure, score saved routes offline and report matched comparisons. Includes a runnable external CPU example and component controls.
-
Reproducible Notebooks
Four notebooks: an evaluator walkthrough, historical validation results, the classical baselines and the results format. Open the sources to view or run them.
-
API Reference
Auto-generated docstrings for the public symbols of
passage_eval.protocol,passage_eval.execution,passage_eval.metrics,passage_eval.diagnostics,passage_eval.trustworthiness,passage_eval.deploymentandpassage_eval.sidecar.
Why a separate package?¶
The dataset and the evaluation contract change at different rates: the released data is frozen, while the evaluation rules can grow as the community adds trustworthiness or deployment checks. passage-eval therefore lives in its own Python package, passage_eval, but ships in the same passage distribution as the generator: uv sync --frozen (or pip install -e .) from the repository root installs both.
You should reach for passage-eval when:
- You have a planner and need decoded-path scores on qualified tasks under an identified procedure.
- You want to compare planners on identical frozen tasks, with intervals that resample whole geographic blocks and with every failure counted. The paper's mean-difference and multiple-testing functions are separate from this comparison workflow.
- You want a reproducibility sidecar for every reported number so reviewers can map any score back to the exact dataset version, seeds, calibration decisions, and hardware stack that produced it.
You do not need passage-eval if you only want to load the dataset (see the Dataset page) or generate samples (see Dataset Generation).
Approach scope¶
The cross-approach evaluation protocol is planner-family-agnostic by design: every primary metric scores the decoded path on the same task and objective. The exact A* reference supplies the cost and geometry comparators. Intermediate outputs such as logits, value fields and policies are useful for diagnostics.
The paper results cover two dense-vision finalists (path-mask segmentation and cost-to-go regression) and a physical A* reference. They are exploratory after prior exposure to the public test split and separate from leaderboard admission. Graph-neural-network and reinforcement-learning planners can use the same contract; the paper names those evaluations as future work.
Source-data acknowledgement¶
passage-eval is licensed under the project's MIT License. The benchmark itself, however, is built on the JAXA ALOS World 3D 30 m DSM (AW3D30); any user of PASSAGE or its derived artifacts must respect JAXA's terms of use — see the JAXA license notice on the home page. DATASET_LICENSE.md in the repository root summarizes the terms; the authoritative text is the JAXA Terms of Use of Research Data.