Skip to content

API Reference

Auto-generated documentation from PASSAGE source code docstrings.


Package Overview

passage

PASSAGE: Path Solving Dataset and Generator.


Core Modules

passage.generate — Dataset Generation

With the shipped config/config.yaml, run_generate runs the generator that built PASSAGE v1.0; its labels come from passage.reference below. The other functions listed here form a sample-by-sample generation path that the v1.0 release does not use.

generate

Generate module for PASSAGE.

run_generate runs the generator that built PASSAGE v1.0 when the configuration sets generation.dataset_version, as the shipped config/config.yaml does. Otherwise it runs a sample-by-sample path that the v1.0 release does not use.

Tiles inside config/hold_out_mask.png (181x361 pixels, one pixel per degree, latitude -90 to +90, longitude -180 to +180) are used only by the test split.

Functions
find_path(crop, start, end, solver_config, prebuilt_graph=None, blocked_mask=None)

Find a path between two points on the crop using configured solver backend.

PARAMETER DESCRIPTION
crop

2D elevation array (not normalized).

TYPE: ndarray

start

(i, j) start coordinates.

TYPE: tuple[int, int]

end

(i, j) end coordinates.

TYPE: tuple[int, int]

solver_config

Solver configuration dict with: - backend: 'networkx' (default), 'pathfinding', or 'grid' - name: solver name for selected backend - diagonal_movement: 'always' (default) or 'never' - weight: float multiplier for elevation cost (default 1.0) - heuristic: 'euclidean' (default), 'manhattan', 'chebyshev' (for astar/best_first) - cost_model: 'elevation' (default), 'distance', 'slope_uphill', 'energy', 'absolute_elevation'

TYPE: dict

prebuilt_graph

Optional prebuilt NetworkX DiGraph. If provided and the selected backend is 'networkx', the prebuilt graph will be used and graph construction will be skipped.

TYPE: DiGraph | None DEFAULT: None

blocked_mask

Optional boolean mask where True marks impassable cells.

TYPE: ndarray | None DEFAULT: None

RETURNS DESCRIPTION
list[tuple[int, int]]

List of (i, j) tuples representing the path, or empty list if no path found.

load_tile(tiles_dirpath, lat, lon)

Load a DSM tile for the given lat/lon coordinates.

PARAMETER DESCRIPTION
tiles_dirpath

Directory containing DSM tiles.

TYPE: Path

lat

Latitude in degrees.

TYPE: int

lon

Longitude in degrees.

TYPE: int

RETURNS DESCRIPTION
ndarray

Numpy array of elevation data (TILE_SIZE_PX x TILE_SIZE_PX).

ndarray

Returns zeros array if tile not found.

load_tile_grid(tiles_dirpath, center_lat, center_lon, half_tiles=1, return_loaded_tiles=False)

Load a (2*half_tiles+1) x (2*half_tiles+1) grid of tiles centered on the given coordinates.

Handles latitude bounds by substituting flat tiles for out-of-bounds rows. Handles longitude wrapping at -180/180.

RETURNS DESCRIPTION
ndarray | tuple[ndarray, list[tuple[int, int]]]
  • grid: concatenated numpy array of shape (NTILE_SIZE_PX, NTILE_SIZE_PX)
ndarray | tuple[ndarray, list[tuple[int, int]]]
  • loaded_tiles: list of (lat, lon) tuples in row-major order corresponding to each tile block
tiles_needed_for_resolution(resolution)

Compute how many tiles are concatenated to produce a crop for the given resolution.

Returns a tuple of
  • N: number of tiles along one axis (2*half_tiles + 1)
  • total_tiles: total tiles concatenated (N*N)
  • half_tiles: number of tiles on each side of the center tile

Example: for small resolutions half_tiles==1 -> N=3 -> total_tiles=9

generate_sample(args)

Generate a single dataset sample.

PARAMETER DESCRIPTION
args

Tuple of (index, tiles_dirpath, outputs_dirpath, resolution, tile_stats_df, global_min, global_max, disable_images, solvers_configs, obstacle_settings, split_name[, generation_seed[, save_preview_images]]).

TYPE: tuple[int, Path, Path, int, DataFrame, float, float, bool, list, dict, str] | tuple[int, Path, Path, int, DataFrame, float, float, bool, list, dict, str, int] | tuple[int, Path, Path, int, DataFrame, float, float, bool, list, dict, str, int, bool]

RETURNS DESCRIPTION
tuple[int, bool, str, float]

Tuple of (index, success, message, duration_seconds).

count_valid_samples(res_dir, disable_images=False, split_assignments=None, preview_image_indices_by_split=None, solver_names=None)

Count consecutive valid samples starting from index 0.

A sample is valid if it has all required files under the expected split/lat/lon subdirectories: - {split}/tensors/{lat}/{lon}/{index:010d}.npy.zst - {split}/metadata/{lat}/{lon}/{index:010d}.yaml - {split}/images/{solver}/{lat}/{lon}/{index:010d}.png for preview-subset rows when images are enabled

Stops counting at the first incomplete sample and deletes any partial files.

PARAMETER DESCRIPTION
res_dir

Resolution directory containing split subdirs.

TYPE: Path

disable_images

If True, don't check for images.

TYPE: bool DEFAULT: False

split_assignments

Mapping from sample index -> split name. If None, assumes a layout without split directories.

TYPE: dict[int, str] | None DEFAULT: None

preview_image_indices_by_split

Optional mapping from split name to the set of sample indices that are expected to have preview images.

TYPE: dict[str, set[int]] | None DEFAULT: None

solver_names

Optional solver names expected under images/<solver>/...

TYPE: list[str] | None DEFAULT: None

RETURNS DESCRIPTION
int

Number of consecutive valid samples from index 0.

run_generate(outputs_dirpath, config_filepath, tiles_dirpath, calibration_filepath='outputs/calibrate/calibrate.json', resolution=None, num_samples=None, num_cpus=None, disable_images=False, force=False, scene_indices=None)

Main orchestration function for generating dataset samples.

PARAMETER DESCRIPTION
outputs_dirpath

Directory path for output files.

TYPE: str

config_filepath

Path to config.yaml file.

TYPE: str

tiles_dirpath

Directory containing DSM tiles.

TYPE: str

calibration_filepath

Path to calibration JSON file.

TYPE: str DEFAULT: 'outputs/calibrate/calibrate.json'

resolution

Image resolution (single int) or list of resolutions.

TYPE: int | list[int] | None DEFAULT: None

num_samples

Number of samples to generate.

TYPE: int | None DEFAULT: None

num_cpus

Number of CPUs to use for multiprocessing.

TYPE: int | None DEFAULT: None

disable_images

If True, skip saving PNG images.

TYPE: bool DEFAULT: False

force

If True, delete existing directories and regenerate all samples; the release generator refuses to overwrite an existing output root.

TYPE: bool DEFAULT: False

scene_indices

Global indices for a bounded release batch; preserve the full sample count.

TYPE: range | list[int] | None DEFAULT: None


passage.reference — Exact Reference Solver

The exact solver that labels every PASSAGE v1.0 route variant with an optimal route, and the connectivity check (endpoints_connected) with which the generator rejects start and goal cells that the obstacles disconnect.

reference

Exact, deterministic implicit-grid references and directed costs-to-go.

No explicit edge graph is built. An indexed binary heap holds at most one entry per cell, with (priority, flattened row-major cell ID) ties. All terrain arithmetic and accumulation are float64, after decoding persisted terrain. The existing GraphSpec destination-only diagonal policy is preserved. Point results are recosted in forward path order; reverse fields accumulate from the goal. Their costs agree within GraphSpec's rtol/atol, with no bitwise equality promise between these summation orders.

Functions
solve_reference(elevation, obstacles, start, goal, *, graph=None, max_expansions=None, algorithm='astar')

Return an exact optimum, proved no_path, or an explicitly censored solve.

Invalid inputs or float64 cost overflow raise ValueError. Blocked endpoints yield proved no_path; arithmetic failure never proves infeasibility. Zero budget censors a distinct-endpoint request. Budgets count expanded outgoing neighborhoods; settling a goal does not spend an expansion. JIT compilation is included in elapsed_seconds on the first invocation.

endpoints_connected(obstacles, start, goal)

Return whether the reference graph has any route from start to goal.

The edge set is the one solve_reference searches: eight neighbours, obstacles checked at the destination cell only, every other edge finite. Reachability therefore needs no terrain, and False is exactly the case the solver would prove no_path. Blocked endpoints are disconnected.

reverse_goal_field(elevation, obstacles, goal, *, graph=None, max_expansions=None)

Dijkstra on transposed edges computes directed original costs to goal.

Relaxing predecessor u from current v adds w(u,v), never w(v,u). Only proved/settled costs are published; targets are evaluator-only. Float64 cost overflow raises ValueError instead of publishing a field.

Re-cost legal integer 8-connected paths; return NaN for illegal routes.


passage.dataset — Bounded Dataset Access

dataset

Bounded, projected access to core and preview PASSAGE Parquet populations.

A local source never falls back to the network. Remote access is explicit and requires a revision; receipts remain unverified until source content is checked.

Classes
DatasetSource(root=None, repo='thalesgroup/passage', revision=None)

Select a local snapshot or a pinned Hugging Face source explicitly.

Functions
batch_size_for(resolution, columns=None, requested=None) staticmethod

Cap tensor-bearing batches near 48 MiB, with one row at 4096².

iter_rows(resolution, split, columns=None, samples=False, batch_size=None, limit=None)

Yield a deterministic prefix; project before reading local/remote batches.

Default columns exclude preview images. limit=None explicitly means the complete selected split; callers producing pilots must supply a cap.

split_manifest(resolution, samples=False)

Census four split headers without reading rows or hashing file contents.

Missing splits and malformed/inconsistent schemas are explicit. Expected release cardinalities require separately pinned configuration evidence; this header census never infers them from preview counts.

get_row(resolution, split, sample_id=None, columns=None, samples=False)

Return the first row or match an ID without decoding preceding tensors.

ID lookup scans only IDs, then reads the matching bounded prefix. Use a stored roster for repeated lookups in large populations.

split_counts(resolution, samples=False)

Read local Parquet footers or remote declared counts, never tensors.

Functions
decode_scene_inputs(row, *, resolution, split)

Validate/decode ordinary tensor inputs independently of stored route labels.


passage.pathfinding_utils — Solvers & Graphs

Graph and collision helpers for the generator's optional solver backends (networkx, pathfinding and grid), which the v1.0 release does not use.

pathfinding_utils

Pathfinding utilities and helpers for PASSAGE.

Functions
build_graph_from_elevation(elevation, weight=1.0, diagonal_movement='always', cost_model='elevation', blocked_mask=None)

Build a directed graph from a 2D elevation array for pathfinding.

Each pixel becomes a node with coordinates (i, j). Edges connect to neighboring pixels with weights based on physical distance and elevation cost.

Edge weight formula depends on cost_model

elevation: distance_m * (1 + weight * normalized_elevation) distance: distance_m slope_uphill: distance_m * (1 + weight * max(0, delta_elev) / elev_range) energy: distance_m + weight * max(0, delta_elev) absolute_elevation: distance_m * (1 + weight * normalized_abs_elevation)

where distance_m is 30m for cardinal directions, 30*sqrt(2) for diagonals.

PARAMETER DESCRIPTION
elevation

2D elevation array (not normalized).

TYPE: ndarray

weight

Multiplier for elevation cost (higher = prefer flat paths).

TYPE: float DEFAULT: 1.0

diagonal_movement

'always' for 8-connectivity, 'never' for 4-connectivity.

TYPE: str DEFAULT: 'always'

cost_model

edge cost model name.

TYPE: str DEFAULT: 'elevation'

RETURNS DESCRIPTION
DiGraph

NetworkX DiGraph with nodes at (i, j) and weighted edges to neighbors.

check_path_collision(path, obstacles_mask)

Check if a path crosses any obstacles in the provided mask.

PARAMETER DESCRIPTION
path

List of (row, col) tuples representing the path.

TYPE: list[tuple[int, int]]

obstacles_mask

2D numpy array where 1 indicates an obstacle and 0 indicates free space.

TYPE: ndarray

RETURNS DESCRIPTION
bool

True if any point on the path coincides with an obstacle, False otherwise.


passage.obstacles — Superellipse Generation

obstacles

Obstacle utilities: superellipse mask generation.

This module implements a small helper to create a binary mask for a rotated superellipse (Lamé curve) given parameters. It documents the meaning of n and provides a reliable implementation that accepts angles in degrees or radians.

Superellipse equation (centered coordinates): (|x|/a)^n + (|y|/b)^n <= 1

Notes about n: - n = 2.0 -> ellipse (circle when a == b) - n < 2 -> "diamond"/pointed shapes (n = 1 -> diamond / rhombus) - n > 2 -> squarer shapes; as n -> large the curve approaches a rectangle - n must be > 0.0 to produce a closed shape

The function here keeps the implementation small and deterministic so it can be used in tests and notebooks. Higher-level obstacle-generation routines can sample n from the configuration to create varied collections of shapes.

Functions
superellipse_mask(shape, center, a, b, n, angle=0.0, angle_degrees=True)

Return a boolean mask of a rotated superellipse.

PARAMETER DESCRIPTION
shape

(rows, cols) of the output mask.

TYPE: Tuple[int, int]

center

(y, x) center coordinates in pixel space (can be floats).

TYPE: Tuple[float, float]

a

semi-axis in x-direction (pixels). Must be > 0.

TYPE: float

b

semi-axis in y-direction (pixels). Must be > 0.

TYPE: float

n

superellipse exponent. Must be > 0. Larger values -> closer to rectangle.

TYPE: float

angle

rotation angle in degrees if angle_degrees is True, otherwise radians.

TYPE: float DEFAULT: 0.0

angle_degrees

interpret angle as degrees when True.

TYPE: bool DEFAULT: True

RETURNS DESCRIPTION
ndarray

Boolean NumPy array with True inside the superellipse and False elsewhere.

RAISES DESCRIPTION
ValueError

if parameters are invalid (non-positive a/b/n or shape not 2-tuple).

sample_n_from_config(obstacles_cfg, rng)

Sample superellipse exponent n from configuration.

Supports the following distribution types in obstacles_cfg['shape']['n']['distribution']: - 'uniform' : uniform between n_min and n_max - 'log_uniform' : uniform in log-space between n_min and n_max - 'beta' : beta distribution (a,b) scaled to [n_min, n_max]

If the nested shape.n structure is missing, the function reads obstacles_cfg['n_min'], obstacles_cfg['n_max'], and obstacles_cfg['n_distribution'].

sample_angle_from_config(rng)

Return an angle in radians uniformly sampled from [0, 2*pi).

Angles are not configurable; they are always sampled uniformly in radians.


passage.calibrate — Elevation Calibration

calibrate

Compute elevation calibration bound to a complete local DSM inventory.

Functions
run_calibrate(outputs_dirpath, tiles_dirpath, *, resume=True, overwrite=False, num_cpus=None)

Serialize calibration checks, source scans, and publication per output directory.

A complete receipt is replaced only by explicit resume=False, overwrite=True. Every complete version remains available by receipt hash.


passage.download — Tile Download

download

Download module for PASSAGE.

This module downloads ALOS AW3D30 tiles by checking URL availability, tracks the download status in a CSV file, and generates a PNG status map.

Functions
download_tile(args)

Download an archive with bounded retries and a hashed completion receipt.

extracted means every DSM member passed CRC/size verification and was published atomically as an individual file. Failures and HTTP absence remain distinct in the archive receipt; the function returns five fields.

generate_status_map(df_results, output_path)

Generate a PNG visualization of URL status.

PARAMETER DESCRIPTION
df_results

DataFrame with lat_min, lon_min, url, url_valid columns.

TYPE: DataFrame

output_path

Path to save the PNG file.

TYPE: Path

run_download(outputs_dirpath, *, resume=True, num_cpus=None, archive_pairs=None)

Download the world grid or explicitly refresh selected 5-degree archives.

Resume validates hashed receipts. CSV records without a hashed receipt remain legacy_unverified without being redownloaded automatically. Explicit archive_pairs retries only those archives, preserving other records and tiles. Use resume=False to force refresh even verified archives. Unverified or failed records never count as complete coverage.


passage.export — Parquet Export & HF Upload

export

Build PASSAGE release artifacts for compression, export, and upload stages.

Classes
ExportSteps(force_all=False, force_notebooks=False, force_assets=False, force_calibration=False, force_analysis_notebooks=False, force_metadata=False, force_readme=False, force_gitattributes=False, force_strip_outputs=False, skip_strip_outputs=False, skip_gitattributes=False, skip_validate_croissant=False, skip_validate_readme=False, skip_audit_compression=False, sample_size_warn_mb=DEFAULT_SAMPLE_SIZE_WARN_MB, parquet_size_warn_mb=DEFAULT_PARQUET_SIZE_WARN_MB) dataclass

Per-step skip/force flags for prepare_export.

Functions
parse_path_csv(csv_path, *, expected_sha256=None)

Parse a path .csv file with rows "row,col" (no header) and return [[row, col], ...].

compress_release(dataset_dirpath='outputs/dataset', calibrate_dirpath='outputs/calibrate', calibration_filepath=None, notebooks_dirpath=None, config_filepath='config/config.yaml', export_dirpath=None, resolutions=None, num_cpus=None)

Generate Parquet shards for the selected resolutions.

prepare_export(dataset_dirpath='outputs/dataset', calibrate_dirpath='outputs/calibrate', calibration_filepath=None, notebooks_dirpath='notebooks', config_filepath='config/config.yaml', export_dirpath=None, resolutions=None, repo_id=None, execute_notebooks=True, max_notebook_retries=3, notebook_retry_delay=5, num_cpus=None, steps=None, keep_existing_configs=False, notebook_cell_timeout=300, notebook_job_timeout=1800, dataset_revision=None, result_root=None)

Prepare the export directory for upload.

Idempotent by default: each step inspects the current export directory and skips work that already produced its expected artifact. Use ExportSteps.force_* flags (or the master force_all) to redo a specific step. Notebook execution covers all four templates at each selected resolution. Outputs are reused only when their input/source/config identities and execution evidence match.

keep_existing_configs also retains compatible declarations from the existing root README. Retain that card only after verifying earlier remote uploads; this option does not contact the repository or verify that those Parquets exist remotely.

upload_export(calibrate_dirpath='outputs/calibrate', calibration_filepath=None, export_dirpath=None, resolutions=None, repo_id=None, env_path=None, secret_path=None, token=None, private=False, create_repo=True, num_cpus=None)

Upload the prepared export tree to Hugging Face.


passage.utils — Shared Utilities

utils

Shared utilities for PASSAGE.

Functions
encode_tensor_blob(arr, level=TENSOR_COMPRESSION_LEVEL)

Encode a numpy array as zstd-compressed raw bytes.

PARAMETER DESCRIPTION
arr

Numpy array to encode.

TYPE: ndarray

level

Zstd compression level (1-22, default TENSOR_COMPRESSION_LEVEL = 12).

TYPE: int DEFAULT: TENSOR_COMPRESSION_LEVEL

RETURNS DESCRIPTION
bytes

Compressed bytes blob.

decode_tensor_blob(blob, shape, dtype='float32')

Decode a zstd-compressed tensor blob back to numpy array.

PARAMETER DESCRIPTION
blob

Zstd-compressed raw bytes.

TYPE: bytes

shape

Target shape for the array.

TYPE: tuple

dtype

Numpy dtype string (default 'float32').

TYPE: str DEFAULT: 'float32'

RETURNS DESCRIPTION
ndarray

Decoded numpy array with specified shape and dtype.

save_tensor(tensor, path, level=TENSOR_COMPRESSION_LEVEL)

Save a numpy tensor to a compressed file.

PARAMETER DESCRIPTION
tensor

Numpy array to save.

TYPE: ndarray

path

File path (should end with .npy.zst).

TYPE: Path

level

Zstd compression level (default TENSOR_COMPRESSION_LEVEL = 12).

TYPE: int DEFAULT: TENSOR_COMPRESSION_LEVEL

load_tensor(path, shape, dtype='float32')

Load a tensor from a compressed file.

PARAMETER DESCRIPTION
path

File path to the compressed tensor.

TYPE: Path

shape

Expected shape of the tensor.

TYPE: tuple

dtype

Numpy dtype string (default 'float32').

TYPE: str DEFAULT: 'float32'

RETURNS DESCRIPTION
ndarray

Decoded numpy array.


passage.visualization — Image Rendering

visualization

Raster previews compatible with OpenCV's HWC uint8 BGR convention.

Functions
tensor_to_image(tensor, path_pixels=None, alt_path_pixels=None, elev_range=None)

Convert HWC elevation/markers[/obstacles] to an HWC uint8 BGR preview.

Both route arguments contain ordered integer (row, column) vertices; the first is blue and the alternative orange, with white halos. Coordinates must be finite integers within the raster. Invalid geometry raises instead of being clipped. This is a display validation, not a route legality check.

Elevation uses the shared sage–sand scale; elev_range is expressed in the same units as channel zero. Unlabelled raster previews may use normalized elevation. For labelled metre plots, use passage.plotting.decode_scene. Obstacles are charcoal with light hatching. White circles/stars mark start and goal, above all paths. The call signature and BGR output are preserved.