API Reference¶
Auto-generated documentation from PASSAGE source code docstrings.
Package Overview¶
passage ¶
PASSAGE: Path Solving Dataset and Generator.
Core Modules¶
passage.generate — Dataset Generation¶
With the shipped config/config.yaml, run_generate runs the generator that built PASSAGE v1.0; its labels come from passage.reference below. The other functions listed here form a sample-by-sample generation path that the v1.0 release does not use.
generate ¶
Generate module for PASSAGE.
run_generate runs the generator that built PASSAGE v1.0 when the configuration sets generation.dataset_version, as the shipped config/config.yaml does. Otherwise it runs a sample-by-sample path that the v1.0 release does not use.
Tiles inside config/hold_out_mask.png (181x361 pixels, one pixel per degree, latitude -90 to +90, longitude -180 to +180) are used only by the test split.
Functions¶
find_path(crop, start, end, solver_config, prebuilt_graph=None, blocked_mask=None) ¶
Find a path between two points on the crop using configured solver backend.
| PARAMETER | DESCRIPTION |
|---|---|
crop | 2D elevation array (not normalized). TYPE: |
start | (i, j) start coordinates. TYPE: |
end | (i, j) end coordinates. TYPE: |
solver_config | Solver configuration dict with: - backend: 'networkx' (default), 'pathfinding', or 'grid' - name: solver name for selected backend - diagonal_movement: 'always' (default) or 'never' - weight: float multiplier for elevation cost (default 1.0) - heuristic: 'euclidean' (default), 'manhattan', 'chebyshev' (for astar/best_first) - cost_model: 'elevation' (default), 'distance', 'slope_uphill', 'energy', 'absolute_elevation' TYPE: |
prebuilt_graph | Optional prebuilt NetworkX DiGraph. If provided and the selected backend is 'networkx', the prebuilt graph will be used and graph construction will be skipped. TYPE: |
blocked_mask | Optional boolean mask where True marks impassable cells. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
list[tuple[int, int]] | List of (i, j) tuples representing the path, or empty list if no path found. |
load_tile(tiles_dirpath, lat, lon) ¶
Load a DSM tile for the given lat/lon coordinates.
| PARAMETER | DESCRIPTION |
|---|---|
tiles_dirpath | Directory containing DSM tiles. TYPE: |
lat | Latitude in degrees. TYPE: |
lon | Longitude in degrees. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
ndarray | Numpy array of elevation data (TILE_SIZE_PX x TILE_SIZE_PX). |
ndarray | Returns zeros array if tile not found. |
load_tile_grid(tiles_dirpath, center_lat, center_lon, half_tiles=1, return_loaded_tiles=False) ¶
Load a (2*half_tiles+1) x (2*half_tiles+1) grid of tiles centered on the given coordinates.
Handles latitude bounds by substituting flat tiles for out-of-bounds rows. Handles longitude wrapping at -180/180.
| RETURNS | DESCRIPTION |
|---|---|
ndarray | tuple[ndarray, list[tuple[int, int]]] |
|
ndarray | tuple[ndarray, list[tuple[int, int]]] |
|
tiles_needed_for_resolution(resolution) ¶
Compute how many tiles are concatenated to produce a crop for the given resolution.
Returns a tuple of
- N: number of tiles along one axis (2*half_tiles + 1)
- total_tiles: total tiles concatenated (N*N)
- half_tiles: number of tiles on each side of the center tile
Example: for small resolutions half_tiles==1 -> N=3 -> total_tiles=9
generate_sample(args) ¶
Generate a single dataset sample.
| PARAMETER | DESCRIPTION |
|---|---|
args | Tuple of (index, tiles_dirpath, outputs_dirpath, resolution, tile_stats_df, global_min, global_max, disable_images, solvers_configs, obstacle_settings, split_name[, generation_seed[, save_preview_images]]). TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
tuple[int, bool, str, float] | Tuple of (index, success, message, duration_seconds). |
count_valid_samples(res_dir, disable_images=False, split_assignments=None, preview_image_indices_by_split=None, solver_names=None) ¶
Count consecutive valid samples starting from index 0.
A sample is valid if it has all required files under the expected split/lat/lon subdirectories: - {split}/tensors/{lat}/{lon}/{index:010d}.npy.zst - {split}/metadata/{lat}/{lon}/{index:010d}.yaml - {split}/images/{solver}/{lat}/{lon}/{index:010d}.png for preview-subset rows when images are enabled
Stops counting at the first incomplete sample and deletes any partial files.
| PARAMETER | DESCRIPTION |
|---|---|
res_dir | Resolution directory containing split subdirs. TYPE: |
disable_images | If True, don't check for images. TYPE: |
split_assignments | Mapping from sample index -> split name. If None, assumes a layout without split directories. TYPE: |
preview_image_indices_by_split | Optional mapping from split name to the set of sample indices that are expected to have preview images. TYPE: |
solver_names | Optional solver names expected under TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
int | Number of consecutive valid samples from index 0. |
run_generate(outputs_dirpath, config_filepath, tiles_dirpath, calibration_filepath='outputs/calibrate/calibrate.json', resolution=None, num_samples=None, num_cpus=None, disable_images=False, force=False, scene_indices=None) ¶
Main orchestration function for generating dataset samples.
| PARAMETER | DESCRIPTION |
|---|---|
outputs_dirpath | Directory path for output files. TYPE: |
config_filepath | Path to config.yaml file. TYPE: |
tiles_dirpath | Directory containing DSM tiles. TYPE: |
calibration_filepath | Path to calibration JSON file. TYPE: |
resolution | Image resolution (single int) or list of resolutions. TYPE: |
num_samples | Number of samples to generate. TYPE: |
num_cpus | Number of CPUs to use for multiprocessing. TYPE: |
disable_images | If True, skip saving PNG images. TYPE: |
force | If True, delete existing directories and regenerate all samples; the release generator refuses to overwrite an existing output root. TYPE: |
scene_indices | Global indices for a bounded release batch; preserve the full sample count. TYPE: |
passage.reference — Exact Reference Solver¶
The exact solver that labels every PASSAGE v1.0 route variant with an optimal route, and the connectivity check (endpoints_connected) with which the generator rejects start and goal cells that the obstacles disconnect.
reference ¶
Exact, deterministic implicit-grid references and directed costs-to-go.
No explicit edge graph is built. An indexed binary heap holds at most one entry per cell, with (priority, flattened row-major cell ID) ties. All terrain arithmetic and accumulation are float64, after decoding persisted terrain. The existing GraphSpec destination-only diagonal policy is preserved. Point results are recosted in forward path order; reverse fields accumulate from the goal. Their costs agree within GraphSpec's rtol/atol, with no bitwise equality promise between these summation orders.
Functions¶
solve_reference(elevation, obstacles, start, goal, *, graph=None, max_expansions=None, algorithm='astar') ¶
Return an exact optimum, proved no_path, or an explicitly censored solve.
Invalid inputs or float64 cost overflow raise ValueError. Blocked endpoints yield proved no_path; arithmetic failure never proves infeasibility. Zero budget censors a distinct-endpoint request. Budgets count expanded outgoing neighborhoods; settling a goal does not spend an expansion. JIT compilation is included in elapsed_seconds on the first invocation.
endpoints_connected(obstacles, start, goal) ¶
Return whether the reference graph has any route from start to goal.
The edge set is the one solve_reference searches: eight neighbours, obstacles checked at the destination cell only, every other edge finite. Reachability therefore needs no terrain, and False is exactly the case the solver would prove no_path. Blocked endpoints are disconnected.
reverse_goal_field(elevation, obstacles, goal, *, graph=None, max_expansions=None) ¶
Dijkstra on transposed edges computes directed original costs to goal.
Relaxing predecessor u from current v adds w(u,v), never w(v,u). Only proved/settled costs are published; targets are evaluator-only. Float64 cost overflow raises ValueError instead of publishing a field.
recost_legal_path(elevation, path, *, graph=None, obstacles=None, start=None, goal=None) ¶
Re-cost legal integer 8-connected paths; return NaN for illegal routes.
passage.dataset — Bounded Dataset Access¶
dataset ¶
Bounded, projected access to core and preview PASSAGE Parquet populations.
A local source never falls back to the network. Remote access is explicit and requires a revision; receipts remain unverified until source content is checked.
Classes¶
DatasetSource(root=None, repo='thalesgroup/passage', revision=None) ¶
Select a local snapshot or a pinned Hugging Face source explicitly.
batch_size_for(resolution, columns=None, requested=None) staticmethod ¶Cap tensor-bearing batches near 48 MiB, with one row at 4096².
iter_rows(resolution, split, columns=None, samples=False, batch_size=None, limit=None) ¶Yield a deterministic prefix; project before reading local/remote batches.
Default columns exclude preview images. limit=None explicitly means the complete selected split; callers producing pilots must supply a cap.
split_manifest(resolution, samples=False) ¶Census four split headers without reading rows or hashing file contents.
Missing splits and malformed/inconsistent schemas are explicit. Expected release cardinalities require separately pinned configuration evidence; this header census never infers them from preview counts.
get_row(resolution, split, sample_id=None, columns=None, samples=False) ¶Return the first row or match an ID without decoding preceding tensors.
ID lookup scans only IDs, then reads the matching bounded prefix. Use a stored roster for repeated lookups in large populations.
split_counts(resolution, samples=False) ¶Read local Parquet footers or remote declared counts, never tensors.
Functions¶
decode_scene_inputs(row, *, resolution, split) ¶
Validate/decode ordinary tensor inputs independently of stored route labels.
passage.pathfinding_utils — Solvers & Graphs¶
Graph and collision helpers for the generator's optional solver backends (networkx, pathfinding and grid), which the v1.0 release does not use.
pathfinding_utils ¶
Pathfinding utilities and helpers for PASSAGE.
Functions¶
build_graph_from_elevation(elevation, weight=1.0, diagonal_movement='always', cost_model='elevation', blocked_mask=None) ¶
Build a directed graph from a 2D elevation array for pathfinding.
Each pixel becomes a node with coordinates (i, j). Edges connect to neighboring pixels with weights based on physical distance and elevation cost.
Edge weight formula depends on cost_model
elevation: distance_m * (1 + weight * normalized_elevation) distance: distance_m slope_uphill: distance_m * (1 + weight * max(0, delta_elev) / elev_range) energy: distance_m + weight * max(0, delta_elev) absolute_elevation: distance_m * (1 + weight * normalized_abs_elevation)
where distance_m is 30m for cardinal directions, 30*sqrt(2) for diagonals.
| PARAMETER | DESCRIPTION |
|---|---|
elevation | 2D elevation array (not normalized). TYPE: |
weight | Multiplier for elevation cost (higher = prefer flat paths). TYPE: |
diagonal_movement | 'always' for 8-connectivity, 'never' for 4-connectivity. TYPE: |
cost_model | edge cost model name. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
DiGraph | NetworkX DiGraph with nodes at (i, j) and weighted edges to neighbors. |
check_path_collision(path, obstacles_mask) ¶
Check if a path crosses any obstacles in the provided mask.
| PARAMETER | DESCRIPTION |
|---|---|
path | List of (row, col) tuples representing the path. TYPE: |
obstacles_mask | 2D numpy array where 1 indicates an obstacle and 0 indicates free space. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
bool | True if any point on the path coincides with an obstacle, False otherwise. |
passage.obstacles — Superellipse Generation¶
obstacles ¶
Obstacle utilities: superellipse mask generation.
This module implements a small helper to create a binary mask for a rotated superellipse (Lamé curve) given parameters. It documents the meaning of n and provides a reliable implementation that accepts angles in degrees or radians.
Superellipse equation (centered coordinates): (|x|/a)^n + (|y|/b)^n <= 1
Notes about n: - n = 2.0 -> ellipse (circle when a == b) - n < 2 -> "diamond"/pointed shapes (n = 1 -> diamond / rhombus) - n > 2 -> squarer shapes; as n -> large the curve approaches a rectangle - n must be > 0.0 to produce a closed shape
The function here keeps the implementation small and deterministic so it can be used in tests and notebooks. Higher-level obstacle-generation routines can sample n from the configuration to create varied collections of shapes.
Functions¶
superellipse_mask(shape, center, a, b, n, angle=0.0, angle_degrees=True) ¶
Return a boolean mask of a rotated superellipse.
| PARAMETER | DESCRIPTION |
|---|---|
shape | (rows, cols) of the output mask. TYPE: |
center | (y, x) center coordinates in pixel space (can be floats). TYPE: |
a | semi-axis in x-direction (pixels). Must be > 0. TYPE: |
b | semi-axis in y-direction (pixels). Must be > 0. TYPE: |
n | superellipse exponent. Must be > 0. Larger values -> closer to rectangle. TYPE: |
angle | rotation angle in degrees if TYPE: |
angle_degrees | interpret TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
ndarray | Boolean NumPy array with True inside the superellipse and False elsewhere. |
| RAISES | DESCRIPTION |
|---|---|
ValueError | if parameters are invalid (non-positive a/b/n or shape not 2-tuple). |
sample_n_from_config(obstacles_cfg, rng) ¶
Sample superellipse exponent n from configuration.
Supports the following distribution types in obstacles_cfg['shape']['n']['distribution']: - 'uniform' : uniform between n_min and n_max - 'log_uniform' : uniform in log-space between n_min and n_max - 'beta' : beta distribution (a,b) scaled to [n_min, n_max]
If the nested shape.n structure is missing, the function reads obstacles_cfg['n_min'], obstacles_cfg['n_max'], and obstacles_cfg['n_distribution'].
sample_angle_from_config(rng) ¶
Return an angle in radians uniformly sampled from [0, 2*pi).
Angles are not configurable; they are always sampled uniformly in radians.
passage.calibrate — Elevation Calibration¶
calibrate ¶
Compute elevation calibration bound to a complete local DSM inventory.
Functions¶
run_calibrate(outputs_dirpath, tiles_dirpath, *, resume=True, overwrite=False, num_cpus=None) ¶
Serialize calibration checks, source scans, and publication per output directory.
A complete receipt is replaced only by explicit resume=False, overwrite=True. Every complete version remains available by receipt hash.
passage.download — Tile Download¶
download ¶
Download module for PASSAGE.
This module downloads ALOS AW3D30 tiles by checking URL availability, tracks the download status in a CSV file, and generates a PNG status map.
Functions¶
download_tile(args) ¶
Download an archive with bounded retries and a hashed completion receipt.
extracted means every DSM member passed CRC/size verification and was published atomically as an individual file. Failures and HTTP absence remain distinct in the archive receipt; the function returns five fields.
generate_status_map(df_results, output_path) ¶
Generate a PNG visualization of URL status.
| PARAMETER | DESCRIPTION |
|---|---|
df_results | DataFrame with lat_min, lon_min, url, url_valid columns. TYPE: |
output_path | Path to save the PNG file. TYPE: |
run_download(outputs_dirpath, *, resume=True, num_cpus=None, archive_pairs=None) ¶
Download the world grid or explicitly refresh selected 5-degree archives.
Resume validates hashed receipts. CSV records without a hashed receipt remain legacy_unverified without being redownloaded automatically. Explicit archive_pairs retries only those archives, preserving other records and tiles. Use resume=False to force refresh even verified archives. Unverified or failed records never count as complete coverage.
passage.export — Parquet Export & HF Upload¶
export ¶
Build PASSAGE release artifacts for compression, export, and upload stages.
Classes¶
ExportSteps(force_all=False, force_notebooks=False, force_assets=False, force_calibration=False, force_analysis_notebooks=False, force_metadata=False, force_readme=False, force_gitattributes=False, force_strip_outputs=False, skip_strip_outputs=False, skip_gitattributes=False, skip_validate_croissant=False, skip_validate_readme=False, skip_audit_compression=False, sample_size_warn_mb=DEFAULT_SAMPLE_SIZE_WARN_MB, parquet_size_warn_mb=DEFAULT_PARQUET_SIZE_WARN_MB) dataclass ¶
Per-step skip/force flags for prepare_export.
Functions¶
parse_path_csv(csv_path, *, expected_sha256=None) ¶
Parse a path .csv file with rows "row,col" (no header) and return [[row, col], ...].
compress_release(dataset_dirpath='outputs/dataset', calibrate_dirpath='outputs/calibrate', calibration_filepath=None, notebooks_dirpath=None, config_filepath='config/config.yaml', export_dirpath=None, resolutions=None, num_cpus=None) ¶
Generate Parquet shards for the selected resolutions.
prepare_export(dataset_dirpath='outputs/dataset', calibrate_dirpath='outputs/calibrate', calibration_filepath=None, notebooks_dirpath='notebooks', config_filepath='config/config.yaml', export_dirpath=None, resolutions=None, repo_id=None, execute_notebooks=True, max_notebook_retries=3, notebook_retry_delay=5, num_cpus=None, steps=None, keep_existing_configs=False, notebook_cell_timeout=300, notebook_job_timeout=1800, dataset_revision=None, result_root=None) ¶
Prepare the export directory for upload.
Idempotent by default: each step inspects the current export directory and skips work that already produced its expected artifact. Use ExportSteps.force_* flags (or the master force_all) to redo a specific step. Notebook execution covers all four templates at each selected resolution. Outputs are reused only when their input/source/config identities and execution evidence match.
keep_existing_configs also retains compatible declarations from the existing root README. Retain that card only after verifying earlier remote uploads; this option does not contact the repository or verify that those Parquets exist remotely.
upload_export(calibrate_dirpath='outputs/calibrate', calibration_filepath=None, export_dirpath=None, resolutions=None, repo_id=None, env_path=None, secret_path=None, token=None, private=False, create_repo=True, num_cpus=None) ¶
Upload the prepared export tree to Hugging Face.
passage.utils — Shared Utilities¶
utils ¶
Shared utilities for PASSAGE.
Functions¶
encode_tensor_blob(arr, level=TENSOR_COMPRESSION_LEVEL) ¶
Encode a numpy array as zstd-compressed raw bytes.
| PARAMETER | DESCRIPTION |
|---|---|
arr | Numpy array to encode. TYPE: |
level | Zstd compression level (1-22, default TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
bytes | Compressed bytes blob. |
decode_tensor_blob(blob, shape, dtype='float32') ¶
Decode a zstd-compressed tensor blob back to numpy array.
| PARAMETER | DESCRIPTION |
|---|---|
blob | Zstd-compressed raw bytes. TYPE: |
shape | Target shape for the array. TYPE: |
dtype | Numpy dtype string (default 'float32'). TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
ndarray | Decoded numpy array with specified shape and dtype. |
save_tensor(tensor, path, level=TENSOR_COMPRESSION_LEVEL) ¶
Save a numpy tensor to a compressed file.
| PARAMETER | DESCRIPTION |
|---|---|
tensor | Numpy array to save. TYPE: |
path | File path (should end with .npy.zst). TYPE: |
level | Zstd compression level (default TYPE: |
load_tensor(path, shape, dtype='float32') ¶
Load a tensor from a compressed file.
| PARAMETER | DESCRIPTION |
|---|---|
path | File path to the compressed tensor. TYPE: |
shape | Expected shape of the tensor. TYPE: |
dtype | Numpy dtype string (default 'float32'). TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
ndarray | Decoded numpy array. |
passage.visualization — Image Rendering¶
visualization ¶
Raster previews compatible with OpenCV's HWC uint8 BGR convention.
Functions¶
tensor_to_image(tensor, path_pixels=None, alt_path_pixels=None, elev_range=None) ¶
Convert HWC elevation/markers[/obstacles] to an HWC uint8 BGR preview.
Both route arguments contain ordered integer (row, column) vertices; the first is blue and the alternative orange, with white halos. Coordinates must be finite integers within the raster. Invalid geometry raises instead of being clipped. This is a display validation, not a route legality check.
Elevation uses the shared sage–sand scale; elev_range is expressed in the same units as channel zero. Unlabelled raster previews may use normalized elevation. For labelled metre plots, use passage.plotting.decode_scene. Obstacles are charcoal with light hatching. White circles/stars mark start and goal, above all paths. The call signature and BGR output are preserved.