AGILab Architecture

This page gives a single place to understand how the repository is organised, which services collaborate at runtime, and where to hook in when building a new app or extending the platform.

New to AGILab? Start with Quick-Start for a first run, use Architecture in 5 minutes for the compact layer map, then return here when you need the big picture of how the layers fit together.

The short version is:

UI / CLI / notebook
  -> AgiEnv
  -> AGI.run / AGI.install / AGI.get_distrib
  -> worker packaging
  -> local, pool, Dask, or SSH execution
  -> artifacts, evidence, ANALYSIS, notebook export

Everything else in this page explains who owns each part of that path.

Three-plane mental model

AGILAB is easiest to understand as three planes kept in one releaseable monorepo:

Plane	Owns	Main roots
Control plane	Product entry points, runtime APIs, environment resolution, worker packaging, and local/distributed execution.	src/agilab/core/*, src/agilab/lib/agi-gui, src/agilab/lib/agi-web, src/agilab/pages
Payload plane	Apps, page bundles, templates, notebooks, examples, and PyPI payload umbrellas.	src/agilab/apps/builtin, src/agilab/apps-pages, src/agilab/lib/agi-apps, src/agilab/lib/agi-pages, src/agilab/examples
Evidence plane	Proof, audits, release contracts, supply-chain evidence, UI robot outputs, docs mirror, and agent/runbook automation.	tools, .github, docs/source, .codex, .claude, badges

This split is deliberate: the same source tree carries the runtime, the reusable app/page payloads, and the evidence proving those paths.

Global AGILAB architecture from entry surfaces to the reusable app contract, runtime back-planes, evidence, portability, and operational guardrails.

Component view

Pyreverse snapshot of how the web interface entry points, agi_core façade, agi_env and agi_cluster exchange data before the workers are started.

Pipeline example

Data flows from trajectory generators and environment maps into simulation stages, decision engines (learned and/or optimization baselines), and finally visualisation and KPI reporting.

agilab.py navigation

agilab.py exposes the landing and SETTINGS entry points plus core workflow pages (PROJECT/ORCHESTRATE/WORKFLOW/ANALYSIS) and optional page bundles, all routed into agi_core for orchestration.

Manager vs worker responsibilities

An app manager prepares arguments and submits a WorkDispatcher plan through AGI.run; the selected local/pool/Dask back-plane executes packaged worker runtimes.

setup.py exists only for the worker-dispatch packaging path: some distributed upload paths still serialize worker code as egg archives. Public PyPI release packaging is handled by pyproject.toml and uv.

Runtime ownership

Role	Main code roots	Contract
User surfaces	src/agilab/pages, src/agilab/examples, tools/run_configs	Capture user intent and translate it into the public AGI.* actions.
Framework helpers	agi_core, agi_gui	Keep pages and CLI mirrors thin: app loading, shared widgets, telemetry, and UI/page helpers.
Environment resolver	agi_env	Build an AgiEnv for the active project: paths, settings, shares, logs, and runtime environment variables.
Execution facade	agi_cluster.agi_distributor.AGI	Expose install, get_distrib, run, and service actions used by both UI and automation.
Worker packaging and runtime	agi_node, ~/wenv/<app>_worker	Package worker code, bootstrap worker environments, and run app-specific stages.
Execution back-plane	local process, pool, Dask, SSH workers	Execute one coarse AGILAB work item per worker and return artifacts, logs, and telemetry.
Apps	src/agilab/apps and packaged agi-app-* payloads	Keep project-specific manager code, worker code, settings, datasets, and examples behind the common app contract.

Package names versus runtime roles

Some names are close but not interchangeable:

• agi_core is the shared app/page framework helper layer. It is not the worker runtime.

• agi_gui contains Streamlit-facing helpers and must stay out of headless worker manifests.

• agi_env resolves the selected app, settings, logs, workspace paths, and environment variables.

• agi_node owns worker bootstrap, worker base classes, and worker packaging.

• agi_cluster exposes the AGI facade and the distributor that dispatches local, pool, Dask, or SSH runs.

• agi-distributor is the documentation/API page name for the distributor implemented under agi_cluster.agi_distributor.

Manager and worker dependency rule

AGILAB is intentionally a two-runtime system. The manager/runtime side resolves settings, UI state, snippets, and orchestration. The worker/runtime side runs the packaged worker code from ~/wenv/<app>_worker.

That split is why manager imports and worker imports are different contracts. A dependency or path can be valid on the manager side and still be missing or packaged differently on the worker side.

Practical rule:

• put UI and orchestration dependencies in the app project manifest

• put compute-stage dependencies in src/<app>_worker/pyproject.toml

• do not put Streamlit or page-only dependencies in worker manifests

• when a worker install fails, compare the source worker manifest with the deployed copy under ~/wenv/<app>_worker before changing app code

Execution back-plane boundary

agi-distributor API contains the Dask-based scheduler, worker templates, and capacity-weighted work-plan balancer. Optional cluster helpers for SSH, remote installs, and zip staging live under src/agilab/core/agi-node and are reused by every app.

AGILAB submits one coarse AGILAB task per worker to the outer Dask scheduler. The code that runs inside BaseWorker.works(...) is intentionally opaque to that outer scheduler. This keeps the worker contract stable across plain local, pool-based, and Dask-based execution modes.

If a worker starts its own inner Dask client or scheduler, the outer Dask/Bokeh dashboard only sees the outer AGILAB worker future, not the inner task graph. Treat Dask as AGILAB’s cluster back-plane for workers, not as a supported inner orchestration engine inside one worker process.

Runtime flow

1. A web button, CLI script, notebook, or run configuration selects an app and prepares run parameters.

2. The entry point instantiates AgiEnv with the selected app. AgiEnv resolves symlinks, copies optional data bundles, seeds ~/.agilab/apps/<app>/app_settings.toml from the app’s versioned source app_settings.toml (<project>/app_settings.toml or <project>/src/app_settings.toml) when needed, and loads overrides from that workspace copy.

3. AGI.run (or AGI.get_distrib / AGI.install) selects the dispatcher mode, builds or reuses the worker wheel, and starts a scheduler locally or on the configured SSH hosts. The manager/runtime process does not execute the worker logic directly; it prepares and dispatches the worker/runtime package.

4. agi-distributor API starts workers, streams WorkDispatcher plans derived from the app manager, and feeds telemetry back into the capacity predictor.

5. Results land in ~/agi-space (for end users) or the repo data/export folders (for developers), while logs are mirrored to ~/log/execute/<app>/ for reproducibility.

Apps choose their own distribution unit. For example, the UAV Relay Queue demo (install id uav_relay_queue_project) fans out one scenario JSON file per worker and writes each run into its own output directory so distributed runs can keep per-scenario artifacts isolated.

Two common execution modes:

• Local notebook / laptop – scheduler + workers run on the same machine. Use this for prototyping and keep an eye on ~/log/execute/<app>/ for telemetry.

• Cluster / SSH hosts – scheduler runs locally, workers spawn remotely via the SSH helpers in agi_cluster.agi_distributor. Provide credentials via ~/.agilab/.env and rerun pycharm/setup_pycharm.py after editing run configurations so CLI wrappers stay synced.

Repository map

Tracked repository tree snapshot

agilab/
__pycache__/ [excluded]
apps/
    builtin/
        flight_telemetry_project/
            .venv/ [excluded]
            agilab/
                core/
                    agi-env/
                        src/
                            agi_env/
                                resources/
                                    .agilab/
                                        [max depth reached]
                                    mistral_offline/
                                        [max depth reached]
            Modules/ [excluded]
            src/
                __pycache__/ [excluded]
                flight_telemetry/
                    __pycache__/ [excluded]
                    __init__.py
                    flight_telemetry.py
                    flight_args.py
                flight_telemetry_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    dataset.7z
                    flight_telemetry_worker.py
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            test/
            .coverage
            .gitignore
            classes_flight.dot
            packages_flight.dot
            pyproject.toml
            README.md
        minimal_app_project/
            .venv/ [excluded]
            agilab/
                core/
                    agi-env/
                        src/
                            agi_env/
                                resources/
                                    .agilab/
                                        [max depth reached]
                                    mistral_offline/
                                        [max depth reached]
            Modules/ [excluded]
            src/
                minimal_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    minimal_app.py
                    minimal_app_args.py
                minimal_app_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    minimal_app_worker.py
                    minimal_app_worker.pyx
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            test/
            .coverage
            .gitignore
            pyproject.toml
            README.md
    templates/
        dag_app_template/
            src/
                __pycache__/ [excluded]
                dag_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    dag_app.py
                    dag_app_args.py
                dag_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        fireducks_app_template/
            src/
                __pycache__/ [excluded]
                fireducks_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    fireducks_app.py
                    fireducks_app_args.py
                fireducks_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        pandas_app_template/
            src/
                __pycache__/ [excluded]
                pandas_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    pandas_app.py
                    pandas_app_args.py
                pandas_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        polars_app_template/
            src/
                __pycache__/ [excluded]
                polars_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    polars_app.py
                    polars_app_args.py
                polars_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
    .gitignore
    __init__.py
    install.py
    README.md
apps-pages/
    view_autoencoder_latentspace/
        .venv/ [excluded]
        src/
            view_autoencoder_latentspace/
                __init__.py
                autoencoder_latentspace.py
                view_autoencoder_latentspace.py
        pyproject.toml
    view_barycentric/
        .venv/ [excluded]
        src/
            view_barycentric/
                __init__.py
                barycentric_graph.py
                view_barycentric.py
        __init__.py
        pyproject.toml
    view_maps/
        .venv/ [excluded]
        src/
            view_maps/
                __pycache__/ [excluded]
                __init__.py
                maps.py
                view_maps.py
        pyproject.toml
    view_maps_3d/
        .venv/ [excluded]
        src/
            view_maps_3d/
                __pycache__/ [excluded]
                __init__.py
                maps_3d.py
                view_maps_3d.py
        pyproject.toml
    view_maps_network/
        .venv/ [excluded]
        src/
            view_maps_network/
                __pycache__/ [excluded]
                maps_network_graph.py
                view_maps_network.py
        pyproject.toml
    .gitignore
    __init__.py
    README.md
core/
    __pycache__/ [excluded]
    agi-cluster/
        .venv/ [excluded]
        __pycache__/ [excluded]
        src/
            agi_cluster/
                agi_distributor/
                    __pycache__/ [excluded]
                    __init__.py
                    agi_distributor.py
                    cli.py
            __init__.py
        .gitignore
        __init__.py
        build.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        uv_config.toml
    agi-core/
        .venv/ [excluded]
        src/
            agi_core/
                __pycache__/ [excluded]
                __init__.py
        LICENSE
        pyproject.toml
        README.md
    agi-env/
        .pytest_cache/ [excluded]
        .venv/ [excluded]
        __pycache__/ [excluded]
        src/
            __pycache__/ [excluded]
            agi_env/
                __pycache__/ [excluded]
                _optional_ui.py
                resources/
                    .agilab/
                        .env
                        balancer_df.csv
                        balancer_model.pkl
                    mistral_offline/
                        data/
                            .gitignore
                            .gitkeep
                        README.md
                __init__.py
                agi_env.py
                agi_logger.py
                app_args.py
                app_settings_support.py
                bootstrap_support.py
                connector_registry.py
                content_renamer_support.py
                credential_store_support.py
                data_archive_support.py
                defaults.py
                env_config_support.py
                execution_support.py
                hook_support.py
                installation_support.py
                mlflow_store.py
                package_layout_support.py
                pagelib.py
                pagelib_data_support.py
                pagelib_execution_support.py
                pagelib_navigation_support.py
                pagelib_preview_support.py
                pagelib_project_support.py
                pagelib_resource_support.py
                pagelib_runtime_support.py
                pagelib_selection_support.py
                pagelib_session_support.py
                process_support.py
                project_clone_support.py
                rename_gitignore_support.py
                repository_support.py
                runtime_bootstrap_support.py
                share_mount_support.py
                share_runtime_support.py
                snippet_contract.py
                source_analysis_ast.py
                source_analysis_support.py
                streamlit_args.py
                ui_docs_support.py
                ui_state_support.py
                ui_support.py
                worker_runtime_support.py
                worker_source_support.py
            __init__.py
            pyproject.toml
        test/
            __pycache__/ [excluded]
            __init__.py
            clean_csv.py
            dummy_cmd.py
            test_agi_env.py
            test_agi_logger.py
            test_app_args.py
            test_app_settings_support.py
            test_bootstrap_support.py
            test_content_renamer_support.py
            test_credential_store_support.py
            test_data_archive_support.py
            test_env_config_support.py
            test_hook_support.py
            test_installation_support.py
            test_package_layout_support.py
            test_pagelib.py
            test_pagelib_data_support.py
            test_pagelib_execution_support.py
            test_pagelib_navigation_support.py
            test_pagelib_preview_support.py
            test_pagelib_project_support.py
            test_pagelib_resource_support.py
            test_pagelib_runtime_support.py
            test_pagelib_session_support.py
            test_process_support.py
            test_project_clone_support.py
            test_rename_gitignore_support.py
            test_repository_support.py
            test_runtime_bootstrap_support.py
            test_share_runtime_support.py
            test_source_analysis_ast.py
            test_source_analysis_support.py
            test_streamlit_args.py
            test_ui_docs_support.py
            test_ui_state_support.py
            test_ui_support.py
            test_worker_source_support.py
        .gitignore
        __init__.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        uv_config.toml
    agi-node/
        .venv/ [excluded]
        src/
            agi_node/
                __pycache__/ [excluded]
                agi_dispatcher/
                    __pycache__/ [excluded]
                    __init__.py
                    agi_dispatcher.py
                    base_worker.py
                    build.py
                    cython_type_preprocess.py
                    post_install.py
                    pre_install.py
                dag_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    dag_worker.py
                fireducks_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    fireducks_worker.py
                pandas_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    pandas_worker.py
                polars_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    polars_worker.py
                __init__.py
                utils.py
            __init__.py
        __init__.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        SECURITY.md
        uv_config.toml
    test/
        __pycache__/ [excluded]
        __init__.py
        test_agi_core_smoke.py
        test_agi_dispatcher_scripts.py
        test_agi_distributor.py
        test_agi_distributor_cli.py
        test_base_worker.py
        test_dag_worker.py
        test_fireducks_worker.py
        test_pandas_worker.py
        test_polars_worker.py
        test_utils.py
        test_work_dispatcher.py
    tools/
        __init__.py
        test_kill.py
    .gitignore
    __init__.py
    gen_app_script.py
    get_supported_python_versions.py
    install.ps1
    install.sh
lib/
    agi-gui/
        src/
            agi_gui/
                __init__.py
                _compat.py
                pagelib.py
                pagelib_data_support.py
                pagelib_execution_support.py
                pagelib_navigation_support.py
                pagelib_preview_support.py
                pagelib_project_support.py
                pagelib_resource_support.py
                pagelib_runtime_support.py
                pagelib_selection_support.py
                pagelib_session_support.py
                streamlit_args.py
                ui_docs_support.py
                ui_state_support.py
                ui_support.py
        test/
            test_agi_gui_package.py
        LICENSE
        pyproject.toml
        README.md
examples/
    flight/
        AGI_install_flight_telemetry.py
        AGI_run_flight_telemetry.py
    minimal_app/
        AGI_install_minimal_app.py
        AGI_run_minimal_app.py
pages/
    __pycache__/ [excluded]
    1_PROJECT.py
    2_ORCHESTRATE.py
    3_WORKFLOW.py
    4_ANALYSIS.py
resources/
    help/
        __pycache__/ [excluded]
        __init__.py
        cluster-help.html
        edit_help.html
        execute_help.html
        experiment_help.html
        index.html
        roadmap.html
        views_help.html
    __init__.py
    agi_logo.png
    agilab_logo.png
    code_editor.scss
    config.toml
    custom_buttons.json
    info_bar.json
    theme.css
test/
    __pycache__/ [excluded]
    __init__.py
    test_model_returns_code.py
    test_python_versions.py
    test_streamlit.sh
.gitignore
__init__.py
main_page.py
agi_codex.py
install_apps.ps1
install_apps.sh
lab_run.py
LICENSE
MANIFEST.in
steps.toml
uv_config.toml

Refresh the tracked tree after repository-layout changes by updating docs/source/directory-structure.txt from a clean checkout.

Core vs optional apps

AGILAB contains the shared runtime plus public built-in seeds; focused agi-app-* and agi-pages payloads make the same app/page contracts installable without depending on a source checkout.

Documentation map

• Quick-Start – install/run instructions and links to sample commands.

• Introduction – background, motivation, and terminology.

• AGI Core Architecture – internals of the web-interface/CLI façade.

• Framework API – reference for AGI.run and the dispatcher helpers.

See also

• Project Files Structure for details on each top-level folder.

• Framework API for the public AGI.* orchestration helpers.

• agi-env API for environment bootstrapping and dataset handling.