AGILab Architecture

This page gives a single place to understand how the repository is organised, which services collaborate at runtime, and where to hook in when building a new app or extending the platform.

New to AGILab? Start with Quick-Start for a first run, use Architecture in 5 minutes for the compact layer map, then return here when you need the big picture of how the layers fit together.

AGILab layers from the web interface down to worker clusters.

Component view

Pyreverse snapshot of how the web interface entry points, agi_core façade, agi_env and agi_cluster exchange data before the workers are started.

Pipeline example

Data flows from trajectory generators and environment maps into simulation stages, decision engines (learned and/or optimization baselines), and finally visualisation and KPI reporting.

agilab.py navigation

agilab.py exposes core pages (PROJECT/ORCHESTRATE/WORKFLOW/ANALYSIS) and optional page bundles (sidecar dashboards), all routed into agi_core for orchestration.

Manager vs worker responsibilities

An app manager prepares arguments and submits plans via AGI.run; workers (BaseWorker subclasses) execute the distributed tasks.

Why setup.py exists: Dask serialises and ships worker code as .egg archives. setup.py is the build hook that generates those archives when the cluster is initialised. It has no role in the PyPI release pipeline — that is handled entirely by pyproject.toml and uv.

Layers at a glance

Layer responsibilities

Layer / Code roots	What it does
User surfaces	Web interface pages (src/agilab/pages) plus CLI mirrors in src/agilab/examples and tools/run_configs. They capture user intent and translate it into AGI.* calls.
agi_core	Reusable widgets, telemetry, app loaders. Keeps web-UI/CLI code thin and prepares WorkDispatcher manifests. See AGI Core Architecture.
agi_env	Discovers datasets, resolves symlinks, stages bundles in ~/wenv and prepares environment variables. Every run builds an AgiEnv first.
agi_cluster / agi_node	Scheduler, workers, balancer, SSH/zip helpers. Turns manifests into Dask jobs locally or on remote hosts.
Apps (``src/agilab/apps``)	Project-specific logic. App managers describe build_distribution and worker arguments but rely on the shared layers above.
Workers (``~/wenv/<app>_worker``)	Cythonised/packaged code deployed on cluster nodes.

User surfaces

• Web interface pages shipped in src/agilab/pages (PROJECT/ORCHESTRATE/WORKFLOW/ANALYSIS).

• CLI mirrors under src/agilab/examples and tools/run_configs.

• tools/run_configs/*.sh can be executed directly from a terminal (no PyCharm required).

• Example scripts in src/agilab/examples (kept in sync via pycharm/setup_pycharm.py).

Core services

• agi-env API handles configuration, environment discovery, dataset staging and artifact caching. Every entry point constructs an AgiEnv before touching a worker.

• agi-node API and agi-distributor API package the reusable logic shared between all apps (dataset helpers, worker bootstrap, git/LFS utilities…).

• Framework API exposes AGI.run / AGI.get_distrib / AGI.install orchestration.

• Apps under src/agilab/apps stay isolated but consume the same base worker / dispatcher APIs. The repository includes example app templates such as mycode_project, flight_project, and the lightweight UAV Relay Queue demo (install id uav_relay_queue_project); additional templates can follow the same contract.

• AGILAB is intentionally a two-runtime system:

  • the manager/runtime side resolves settings, UI state, snippets, and orchestration

  • the worker/runtime side runs the packaged worker code from ~/wenv/<app>_worker

  This split is why manager imports and worker imports are different contracts. A dependency or path can be valid on the manager side and still be missing or packaged differently on the worker side.

Execution back-plane

• agi-distributor API contains the Dask-based scheduler, worker templates and capacity-weighted work-plan balancer. Workers are packaged (python -m agi_node…) into ~/wenv/<app>_worker before run time.

• Optional cluster helpers (SSH, remote installs, zip staging) live under src/agilab/core/agi-node/agi_dispatcher and are reused by every app.

• AGILAB submits one coarse AGILAB task per worker to the outer Dask scheduler. The code that runs inside BaseWorker.works(...) is intentionally opaque to that outer scheduler.

• This boundary is deliberate. It keeps the worker contract stable across plain local, pool-based, and Dask-based execution modes, reduces coupling between app code and Dask internals, and keeps packaging/deployment at one worker-runtime granularity.

• This means nested execution inside a worker is not first-class AGILAB telemetry. If a worker starts its own inner Dask client or scheduler, the outer Dask/Bokeh dashboard only sees the outer AGILAB worker future, not the inner task graph.

• In practice, treat Dask as the cluster back-plane for AGILAB workers, not as the supported inner orchestration engine inside one worker process.

Runtime flow

1. A run configuration (web button, CLI script, PyCharm run config) calls an example in src/agilab/examples/<app>/AGI_run_*.py.

2. The script instantiates AgiEnv with the desired apps_dir and app. AgiEnv resolves symlinks, copies optional data bundles, seeds ~/.agilab/apps/<app>/app_settings.toml from the app’s versioned source app_settings.toml (<project>/app_settings.toml or <project>/src/app_settings.toml) when needed, and loads overrides from that workspace copy.

3. AGI.run (or AGI.get_distrib / AGI.install) selects the dispatcher mode, builds or reuses the worker wheel, and starts a scheduler locally or on the configured SSH hosts. The manager/runtime process does not execute the worker logic directly; it prepares and dispatches the worker/runtime package.

4. agi-distributor API spins up workers, streams WorkDispatcher plans derived from the app manager, and feeds telemetry back into the capacity predictor.

5. Results land in ~/agi-space (for end users) or the repo data/export folders (for developers), while logs are mirrored to ~/log/execute/<app>/ for reproducibility.

Apps choose their own distribution unit. For example, the UAV Relay Queue demo (install id uav_relay_queue_project) fans out one scenario JSON file per worker and writes each run into its own output directory so distributed runs can keep per-scenario artifacts isolated.

Two common execution modes:

• Local notebook / laptop – scheduler + workers run on the same machine. Use this for prototyping and keep an eye on ~/log/execute/<app>/ for telemetry.

• Cluster / SSH hosts – scheduler runs locally, workers spawn remotely via the SSH helpers in agi_cluster.agi_distributor. Provide credentials via ~/.agilab/.env and rerun pycharm/setup_pycharm.py after editing run configurations so CLI wrappers stay synced.

Repository map

Tracked repository tree snapshot

agilab/
__pycache__/ [excluded]
apps/
    builtin/
        flight_project/
            .venv/ [excluded]
            agilab/
                core/
                    agi-env/
                        src/
                            agi_env/
                                resources/
                                    .agilab/
                                        [max depth reached]
                                    mistral_offline/
                                        [max depth reached]
            Modules/ [excluded]
            src/
                __pycache__/ [excluded]
                flight/
                    __pycache__/ [excluded]
                    __init__.py
                    flight.py
                    flight_args.py
                flight_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    dataset.7z
                    flight_worker.c
                    flight_worker.py
                    flight_worker.pyx
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            test/
            .coverage
            .gitignore
            classes_flight.dot
            packages_flight.dot
            pyproject.toml
            README.md
        mycode_project/
            .venv/ [excluded]
            agilab/
                core/
                    agi-env/
                        src/
                            agi_env/
                                resources/
                                    .agilab/
                                        [max depth reached]
                                    mistral_offline/
                                        [max depth reached]
            Modules/ [excluded]
            src/
                mycode/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    mycode.py
                    mycode_args.py
                mycode_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    mycode_worker.py
                    mycode_worker.pyx
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            test/
            .coverage
            .gitignore
            pyproject.toml
            README.md
    templates/
        dag_app_template/
            src/
                __pycache__/ [excluded]
                dag_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    dag_app.py
                    dag_app_args.py
                dag_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        fireducks_app_template/
            src/
                __pycache__/ [excluded]
                fireducks_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    fireducks_app.py
                    fireducks_app_args.py
                fireducks_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        pandas_app_template/
            src/
                __pycache__/ [excluded]
                pandas_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    pandas_app.py
                    pandas_app_args.py
                pandas_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        polars_app_template/
            src/
                __pycache__/ [excluded]
                polars_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    polars_app.py
                    polars_app_args.py
                polars_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
    .gitignore
    __init__.py
    install.py
    README.md
apps-pages/
    view_autoencoder_latenspace/
        .venv/ [excluded]
        src/
            view_autoencoder_latentspace/
                __init__.py
                autoencoder_latentspace.py
                view_autoencoder_latentspace.py
        pyproject.toml
    view_barycentric/
        .venv/ [excluded]
        src/
            view_barycentric/
                __init__.py
                barycentric_graph.py
                view_barycentric.py
        __init__.py
        pyproject.toml
    view_maps/
        .venv/ [excluded]
        src/
            view_maps/
                __pycache__/ [excluded]
                __init__.py
                maps.py
                view_maps.py
        pyproject.toml
    view_maps_3d/
        .venv/ [excluded]
        src/
            view_maps_3d/
                __pycache__/ [excluded]
                __init__.py
                maps_3d.py
                view_maps_3d.py
        pyproject.toml
    view_maps_network/
        .venv/ [excluded]
        src/
            view_maps_network/
                __pycache__/ [excluded]
                maps_network_graph.py
                view_maps_network.py
        pyproject.toml
    .gitignore
    __init__.py
    README.md
core/
    __pycache__/ [excluded]
    agi-cluster/
        .venv/ [excluded]
        __pycache__/ [excluded]
        src/
            agi_cluster/
                agi_distributor/
                    __pycache__/ [excluded]
                    __init__.py
                    agi_distributor.py
                    cli.py
            __init__.py
        .gitignore
        __init__.py
        build.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        uv_config.toml
    agi-core/
        .venv/ [excluded]
        src/
            agi_core/
                __pycache__/ [excluded]
                __init__.py
        LICENSE
        pyproject.toml
        README.md
    agi-env/
        .pytest_cache/ [excluded]
        .venv/ [excluded]
        __pycache__/ [excluded]
        src/
            __pycache__/ [excluded]
            agi_env/
                __pycache__/ [excluded]
                _optional_ui.py
                resources/
                    .agilab/
                        .env
                        balancer_df.csv
                        balancer_model.pkl
                    mistral_offline/
                        data/
                            .gitignore
                            .gitkeep
                        README.md
                __init__.py
                agi_env.py
                agi_logger.py
                app_args.py
                app_settings_support.py
                bootstrap_support.py
                connector_registry.py
                content_renamer_support.py
                credential_store_support.py
                data_archive_support.py
                defaults.py
                env_config_support.py
                execution_support.py
                hook_support.py
                installation_support.py
                mlflow_store.py
                package_layout_support.py
                pagelib.py
                pagelib_data_support.py
                pagelib_execution_support.py
                pagelib_navigation_support.py
                pagelib_preview_support.py
                pagelib_project_support.py
                pagelib_resource_support.py
                pagelib_runtime_support.py
                pagelib_selection_support.py
                pagelib_session_support.py
                process_support.py
                project_clone_support.py
                rename_gitignore_support.py
                repository_support.py
                runtime_bootstrap_support.py
                share_mount_support.py
                share_runtime_support.py
                snippet_contract.py
                source_analysis_ast.py
                source_analysis_support.py
                streamlit_args.py
                ui_docs_support.py
                ui_state_support.py
                ui_support.py
                worker_runtime_support.py
                worker_source_support.py
            __init__.py
            pyproject.toml
        test/
            __pycache__/ [excluded]
            __init__.py
            clean_csv.py
            dummy_cmd.py
            test_agi_env.py
            test_agi_logger.py
            test_app_args.py
            test_app_settings_support.py
            test_bootstrap_support.py
            test_content_renamer_support.py
            test_credential_store_support.py
            test_data_archive_support.py
            test_env_config_support.py
            test_hook_support.py
            test_installation_support.py
            test_package_layout_support.py
            test_pagelib.py
            test_pagelib_data_support.py
            test_pagelib_execution_support.py
            test_pagelib_navigation_support.py
            test_pagelib_preview_support.py
            test_pagelib_project_support.py
            test_pagelib_resource_support.py
            test_pagelib_runtime_support.py
            test_pagelib_session_support.py
            test_process_support.py
            test_project_clone_support.py
            test_rename_gitignore_support.py
            test_repository_support.py
            test_runtime_bootstrap_support.py
            test_share_runtime_support.py
            test_source_analysis_ast.py
            test_source_analysis_support.py
            test_streamlit_args.py
            test_ui_docs_support.py
            test_ui_state_support.py
            test_ui_support.py
            test_worker_source_support.py
        .gitignore
        __init__.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        uv_config.toml
    agi-node/
        .venv/ [excluded]
        src/
            agi_node/
                __pycache__/ [excluded]
                agi_dispatcher/
                    __pycache__/ [excluded]
                    __init__.py
                    agi_dispatcher.py
                    base_worker.py
                    build.py
                    cython_type_preprocess.py
                    post_install.py
                    pre_install.py
                dag_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    dag_worker.py
                fireducks_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    fireducks_worker.py
                pandas_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    pandas_worker.py
                polars_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    polars_worker.py
                __init__.py
                utils.py
            __init__.py
        __init__.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        SECURITY.md
        uv_config.toml
    test/
        __pycache__/ [excluded]
        __init__.py
        test_agi_core_smoke.py
        test_agi_dispatcher_scripts.py
        test_agi_distributor.py
        test_agi_distributor_cli.py
        test_base_worker.py
        test_dag_worker.py
        test_fireducks_worker.py
        test_pandas_worker.py
        test_polars_worker.py
        test_utils.py
        test_work_dispatcher.py
    tools/
        __init__.py
        test_kill.py
    .gitignore
    __init__.py
    gen_app_script.py
    get_supported_python_versions.py
    install.ps1
    install.sh
lib/
    agi-gui/
        src/
            agi_gui/
                __init__.py
                _compat.py
                pagelib.py
                pagelib_data_support.py
                pagelib_execution_support.py
                pagelib_navigation_support.py
                pagelib_preview_support.py
                pagelib_project_support.py
                pagelib_resource_support.py
                pagelib_runtime_support.py
                pagelib_selection_support.py
                pagelib_session_support.py
                streamlit_args.py
                ui_docs_support.py
                ui_state_support.py
                ui_support.py
        test/
            test_agi_gui_package.py
        LICENSE
        pyproject.toml
        README.md
examples/
    flight/
        AGI_install_flight.py
        AGI_run_flight.py
    mycode/
        AGI_install_mycode.py
        AGI_run_mycode.py
pages/
    __pycache__/ [excluded]
    1_PROJECT.py
    2_ORCHESTRATE.py
    3_WORKFLOW.py
    4_ANALYSIS.py
resources/
    help/
        __pycache__/ [excluded]
        __init__.py
        cluster-help.html
        edit_help.html
        execute_help.html
        experiment_help.html
        index.html
        roadmap.html
        views_help.html
    __init__.py
    agi_logo.png
    agilab_logo.png
    code_editor.scss
    config.toml
    custom_buttons.json
    info_bar.json
    theme.css
test/
    __pycache__/ [excluded]
    __init__.py
    test_model_returns_code.py
    test_python_versions.py
    test_streamlit.sh
.gitignore
__init__.py
main_page.py
agi_codex.py
install_apps.ps1
install_apps.sh
lab_run.py
LICENSE
MANIFEST.in
steps.toml
uv_config.toml

Refresh the tracked tree after repository-layout changes by updating docs/source/directory-structure.txt from a clean checkout.

Core vs optional apps

AGILAB (open-source framework) underpins optional apps that can be installed separately. Public docs only cover the open-source layers and built-in apps.

Documentation map

• Quick-Start – install/run instructions and links to sample commands.

• Introduction – background, motivation, and terminology.

• AGI Core Architecture – internals of the web-interface/CLI façade.

• Framework API – reference for AGI.run and the dispatcher helpers.

See also

• Project Files Structure for details on each top-level folder.

• Framework API for the public AGI.* orchestration helpers.

• agi-env API for environment bootstrapping and dataset handling.