Skip to content

Dataset Generation

Use this guide to build PASSAGE scenes from source elevation tiles and prepare them for publication on the Hugging Face Hub. The Generation Process explains how the generator samples crops, places obstacles and solves routes.

To score a planner on the resulting benchmark (decoded-path metrics, statistical discipline, deployment footprint), see the Benchmark section.

JAXA AW3D30 — license obligations

Every PASSAGE scene is built on the JAXA ALOS World 3D–30 m DSM (AW3D30). Any user of PASSAGE or its derived artifacts must respect JAXA's terms of use — cite the AW3D30 source, propagate the attribution downstream, and read the JAXA license notice before redistributing.

Overview

PASSAGE follows a linear, six-step pipeline. Each step is a CLI subcommand that can be run on its own and produces well-defined outputs:

flowchart LR
    A["Download<br>JAXA tiles"] --> B["Calibrate<br>Elevation range"]
    B --> C["Generate<br>Scenes and routes"]
    C --> D["Compress<br>Parquet shards"]
    D --> E["Export<br>README + notebooks + assets"]
    E --> F["Upload<br>HF Hub publish"]

    style A fill:#0891b233,stroke:#0891b2,stroke-width:1.5px
    style B fill:#a78bfa2e,stroke:#8b6fe0,stroke-width:1.5px
    style C fill:#ffb93833,stroke:#d99a1b,stroke-width:1.5px
    style D fill:#0891b233,stroke:#0891b2,stroke-width:1.5px
    style E fill:#a78bfa2e,stroke:#8b6fe0,stroke-width:1.5px
    style F fill:#ff4d5e2e,stroke:#ff4d5e,stroke-width:1.5px
Step Command What it does Main output
1 passage download Downloads the 5°×5° ZIP archives of 1°×1° AW3D30 tiles tiles/*_DSM.tif, download.csv
2 passage calibrate Scans all tiles for the global min/max elevation calibrate.json
3 passage generate Creates scenes: tensors, metadata and six reference routes Dataset tree with one receipt per scene
4 passage compress Writes Parquet shards for the selected resolutions <R>x<R> and <R>x<R>_samples configurations
5 passage export Prepares the upload-ready export tree README, executed notebooks, assets
6 passage upload Publishes the prepared export tree to Hugging Face Hub dataset repository

For the full release, passage regenerate runs steps 3 to 5 in bounded, resumable batches (see Steps 4-6 — Release).

Interrupted runs

Download, calibration and generation keep receipts: re-run the same command and they resume where they stopped. compress replaces configurations only once the new ones are complete, and export skips steps whose outputs are already valid.

Prerequisites

  • Python 3.11+ on Linux
  • uv package manager (recommended) or pip
  • About 463 GB of disk for the 23,993 AW3D30 DSM tiles of the release, plus room for the generated scenes
  • The release hold-out mask before generation
  • A Hugging Face write token if you plan to publish with upload

What's Next?

Installation

Set up your environment, install dependencies, and configure credentials.

Step 1 — Download

Acquire JAXA elevation tiles, with a hashed receipt per archive.

Step 2 — Calibrate

Compute the global elevation range and record the tile inventory.

Step 3 — Generate

Create multi-resolution scenes with exact reference routes.

Steps 4-6 — Release

Compress Parquet shards, refresh the export tree, and publish to Hugging Face Hub.

Configuration

Full reference for config/config.yaml.