AGILab Architecture

This page gives a single place to understand how the repository is organised, which services collaborate at runtime, and where to hook in when building a new app or extending the platform.

New to AGILab? Start with Quick-Start for a five-minute run-through, then return here when you need the big picture of how the layers fit together.

AGILab layers from the web interface down to worker clusters.

Component view

Pyreverse snapshot of how the web interface entry points, agi_core façade, agi_env and agi_cluster exchange data before the workers are started.

Pipeline example

Data flows from trajectory generators and environment maps into simulation stages, decision engines (learned and/or optimization baselines), and finally visualisation and KPI reporting.

agilab.py navigation

agilab.py exposes core pages (PROJECT/ORCHESTRATE/PIPELINE/ANALYSIS) and optional page bundles (sidecar dashboards), all routed into agi_core for orchestration.

Manager vs worker responsibilities

An app manager prepares arguments and submits plans via AGI.run; workers (BaseWorker subclasses) execute the distributed tasks.

Layers at a glance

Layer responsibilities

Layer / Code roots	What it does
User surfaces	Web interface pages (src/agilab/pages) plus CLI mirrors in src/agilab/examples and tools/run_configs. They capture user intent and translate it into AGI.* calls.
agi_core	Reusable widgets, telemetry, app loaders. Keeps web-UI/CLI code thin and prepares WorkDispatcher manifests. See AGI Core Architecture.
agi_env	Discovers datasets, resolves symlinks, stages bundles in ~/wenv and prepares environment variables. Every run builds an AgiEnv first.
agi_cluster / agi_node	Scheduler, workers, balancer, SSH/zip helpers. Turns manifests into Dask jobs locally or on remote hosts.
Apps (``src/agilab/apps``)	Project-specific logic. App managers describe build_distribution and worker arguments but rely on the shared layers above.
Workers (``~/wenv/<app>_worker``)	Cythonised/packaged code deployed on cluster nodes.

User surfaces

• Web interface pages shipped in src/agilab/pages (PROJECT/ORCHESTRATE/PIPELINE/ANALYSIS).

• CLI mirrors under src/agilab/examples and tools/run_configs.

• tools/run_configs/*.sh can be executed directly from a terminal (no PyCharm required).

• Example scripts in src/agilab/examples (kept in sync via pycharm/setup_pycharm.py).

Core services

• agi-env API handles configuration, environment discovery, dataset staging and artifact caching. Every entry point constructs an AgiEnv before touching a worker.

• agi-node API and agi-distributor API package the reusable logic shared between all apps (dataset helpers, worker bootstrap, git/LFS utilities…).

• Framework API exposes AGI.run / AGI.get_distrib / AGI.install orchestration.

• Apps under src/agilab/apps stay isolated but consume the same base worker / dispatcher APIs. The repository includes example app templates such as mycode_project and flight_project; additional templates can follow the same contract.

Execution back-plane

• agi-distributor API contains the Dask-based scheduler, worker templates and capacity balancer. Workers are packaged (python -m agi_node…) into ~/wenv/<app>_worker before run time.

• Optional cluster helpers (SSH, remote installs, zip staging) live under src/agilab/core/agi-node/agi_dispatcher and are reused by every app.

Runtime flow

1. A run configuration (web button, CLI script, PyCharm run config) calls an example in src/agilab/examples/<app>/AGI_run_*.py.

2. The script instantiates AgiEnv with the desired apps_dir and app. AgiEnv resolves symlinks, copies optional data bundles and loads app_settings.toml overrides.

3. AGI.run (or AGI.get_distrib / AGI.install) selects the dispatcher mode, builds or reuses the worker wheel, and starts a scheduler locally or on the configured SSH hosts.

4. agi-distributor API spins up workers, streams WorkDispatcher plans derived from the app manager, and feeds telemetry back into the capacity predictor.

5. Results land in ~/agi-space (for end users) or the repo data/export folders (for developers), while logs are mirrored to ~/log/execute/<app>/ for reproducibility.

Two common execution modes:

• Local notebook / laptop – scheduler + workers run on the same machine. Use this for prototyping and keep an eye on ~/log/execute/<app>/ for telemetry.

• Cluster / SSH hosts – scheduler runs locally, workers spawn remotely via the SSH helpers in agi_cluster.agi_distributor. Provide credentials via ~/.agilab/.env and rerun pycharm/setup_pycharm.py after editing run configurations so CLI wrappers stay synced.

Repository map

Generated by docs/gen-docs.py (gitignore-aware)

agilab/
__pycache__/ [excluded]
apps/
    builtin/
        flight_project/
            .venv/ [excluded]
            Modules/ [excluded]
            src/
                __pycache__/ [excluded]
                flight/
                    __pycache__/ [excluded]
                    __init__.py
                    flight.py
                    flight_args.py
                flight_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    dataset.7z
                    flight_worker.c
                    flight_worker.py
                    flight_worker.pyx
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            test/
            .coverage
            .gitignore
            pyproject.toml
            README.md
        mycode_project/
            .venv/ [excluded]
            Modules/ [excluded]
            src/
                mycode/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    mycode.py
                    mycode_args.py
                mycode_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    mycode_worker.py
                    mycode_worker.pyx
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            test/
            .coverage
            .gitignore
            pyproject.toml
            README.md
    templates/
        dag_app_template/
            src/
                __pycache__/ [excluded]
                dag_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    dag_app.py
                    dag_app_args.py
                dag_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        fireducks_app_template/
            src/
                __pycache__/ [excluded]
                fireducks_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    fireducks_app.py
                    fireducks_app_args.py
                fireducks_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        pandas_app_template/
            src/
                __pycache__/ [excluded]
                pandas_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    pandas_app.py
                    pandas_app_args.py
                pandas_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        polars_app_template/
            src/
                __pycache__/ [excluded]
                polars_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    polars_app.py
                    polars_app_args.py
                polars_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
    .gitignore
    __init__.py
    install.py
    README.md
apps-pages/
    view_autoencoder_latenspace/
        .venv/ [excluded]
        src/
            view_autoencoder_latentspace/
                __init__.py
                autoencoder_latentspace.py
                view_autoencoder_latentspace.py
        pyproject.toml
    view_barycentric/
        .venv/ [excluded]
        src/
            view_barycentric/
                __init__.py
                barycentric_graph.py
                view_barycentric.py
        __init__.py
        pyproject.toml
    view_maps/
        .venv/ [excluded]
        src/
            view_maps/
                __pycache__/ [excluded]
                __init__.py
                maps.py
                view_maps.py
        pyproject.toml
    view_maps_3d/
        .venv/ [excluded]
        src/
            view_maps_3d/
                __pycache__/ [excluded]
                __init__.py
                maps_3d.py
                view_maps_3d.py
        pyproject.toml
    view_maps_network/
        .venv/ [excluded]
        src/
            view_maps_network/
                __pycache__/ [excluded]
                maps_network_graph.py
                view_maps_network.py
        pyproject.toml
    .gitignore
    __init__.py
    README.md
core/
    __pycache__/ [excluded]
    agi-cluster/
        src/
            agi_cluster/
                agi_distributor/
                    __pycache__/ [excluded]
                    __init__.py
                    agi_distributor.py
                    cli.py
            __init__.py
        .gitignore
        __init__.py
        build.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        uv_config.toml
    agi-core/
        src/
            agi_core/
                __pycache__/ [excluded]
                __init__.py
        LICENSE
        pyproject.toml
        README.md
    agi-env/
        .pytest_cache/ [excluded]
        __pycache__/ [excluded]
        src/
            __pycache__/ [excluded]
            agi_env/
                __pycache__/ [excluded]
                resources/
                    .agilab/
                        .env
                        balancer_df.csv
                        balancer_model.pkl
                    mistral_offline/
                        data/
                            .gitignore
                            .gitkeep
                        README.md
                __init__.py
                agi_env.py
                agi_logger.py
                app_args.py
                defaults.py
                pagelib.py
                streamlit_args.py
            __init__.py
            pyproject.toml
        test/
            __pycache__/ [excluded]
            __init__.py
            clean_csv.py
            dummy_cmd.py
            test_agi_env.py
            test_app_args.py
            test_pagelib.py
            test_streamlit_args.py
        .gitignore
        __init__.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        uv_config.toml
    agi-node/
        src/
            agi_node/
                __pycache__/ [excluded]
                agi_dispatcher/
                    __pycache__/ [excluded]
                    __init__.py
                    agi_dispatcher.py
                    base_worker.py
                    build.py
                    post_install.py
                    pre_install.py
                dag_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    dag_worker.py
                fireducks_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    fireducks_worker.py
                pandas_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    pandas_worker.py
                polars_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    polars_worker.py
                __init__.py
                utils.py
            __init__.py
        __init__.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        SECURITY.md
        uv_config.toml
    test/
        __pycache__/ [excluded]
        __init__.py
        test_agi_core_smoke.py
        test_agi_distributor.py
        test_base_worker.py
        test_dag_worker.py
        test_fireducks_worker.py
        test_pandas_worker.py
        test_polars_worker.py
        test_work_dispatcher.py
    tools/
        __init__.py
        test_kill.py
    .gitignore
    __init__.py
    gen_app_script.py
    get_supported_python_versions.py
    install.ps1
    install.sh
examples/
    flight/
        AGI_command_flight.py
        AGI_install_flight.py
        AGI_run_flight.py
    mycode/
        AGI_install_mycode.py
        AGI_run_mycode.py
pages/
    __pycache__/ [excluded]
    1_▶️ PROJECT.py
    2_▶️ ORCHESTRATE.py
    3_▶️ PIPELINE.py
    4_▶️ ANALYSIS.py
resources/
    help/
        __pycache__/ [excluded]
        __init__.py
        cluster-help.html
        edit_help.html
        execute_help.html
        experiment_help.html
        index.html
        roadmap.html
        views_help.html
    __init__.py
    agi_logo.png
    agilab_logo.png
    code_editor.scss
    config.toml
    custom_buttons.json
    info_bar.json
    theme.css
test/
    __pycache__/ [excluded]
    __init__.py
    test_model_returns_code.py
    test_python_versions.py
    test_streamlit.sh
.gitignore
__init__.py
About_agilab.py
agi_codex.py
install_apps.ps1
install_apps.sh
lab_run.py
LICENSE
MANIFEST.in
steps.toml
uv_config.toml

Regenerate the tree after file changes with uv run --group sphinx --dev docs/gen-docs.py.

Core vs optional apps

AGILAB (open-source framework) underpins optional apps that can be installed separately. Public docs only cover the open-source layers and built-in apps.

Documentation map

• Quick-Start – install/run instructions and links to sample commands.

• Introduction – background, motivation, and terminology.

• AGI Core Architecture – internals of the web-interface/CLI façade.

• Framework API – reference for AGI.run and the dispatcher helpers.

See also

• Project Files Structure for details on each top-level folder.

• Framework API for the public AGI.* orchestration helpers.

• agi-env API for environment bootstrapping and dataset handling.