AGILab Architecture

This page gives a single place to understand how the repository is organised, which services collaborate at runtime, and where to hook in when building a new app or extending the platform.

New to AGILab? Start with Quick-Start for a five-minute run-through, then return here when you need the big picture of how the layers fit together.

AGILab layers from the Streamlit front-end down to worker clusters.

Component view

Pyreverse snapshot of how the Streamlit entry points, agi_core façade, agi_env and agi_cluster exchange data before the workers are started.

Pipeline example

Data flows from trajectory generators and environment maps into simulation stages, decision engines (learned and/or optimization baselines), and finally visualisation and KPI reporting.

AGILAB.py navigation

AGILAB.py exposes core PAGES (Execute/Explore/Edit) and APPS-PAGES for each app, all routed into agi_core for orchestration.

Manager vs worker responsibilities

An app manager prepares arguments and submits plans via AGI.run; workers (BaseWorker subclasses) execute the distributed tasks.

Layers at a glance

Layer responsibilities

Layer / Code roots	What it does
User surfaces	Streamlit pages (src/agilab/pages) plus CLI mirrors in src/agilab/examples and tools/run_configs. They capture user intent and translate it into AGI.* calls.
agi_core	Reusable widgets, telemetry, app loaders. Keeps Streamlit/CLI code thin and prepares WorkDispatcher manifests. See AGI Core Architecture.
agi_env	Discovers datasets, resolves symlinks, stages bundles in ~/wenv and prepares environment variables. Every run builds an AgiEnv first.
agi_cluster / agi_node	Scheduler, workers, balancer, SSH/zip helpers. Turns manifests into Dask jobs locally or on remote hosts.
Apps (``src/agilab/apps``)	Project-specific logic. App managers describe build_distribution and worker arguments but rely on the shared layers above.
Workers (``~/wenv/<app>_worker``)	Cythonised/packaged code deployed on cluster nodes.

User surfaces

• Streamlit pages shipped in src/agilab/pages (EXECUTE/EXPLORE/EDIT/…).

• CLI mirrors under docs/source/apps-pages and tools/run_configs.

• Example scripts in src/agilab/examples (kept in sync via pycharm/setup_pycharm.py).

Core services

• agi-env API handles configuration, environment discovery, dataset staging and artifact caching. Every entry point constructs an AgiEnv before touching a worker.

• agi-node API and agi-distributor API package the reusable logic shared between all apps (dataset helpers, worker bootstrap, git/LFS utilities…).

• Framework API exposes AGI.run / AGI.get_distrib / AGI.install orchestration.

• Apps under src/agilab/apps stay isolated but consume the same base worker / dispatcher APIs. The repository includes example app templates such as mycode_project and flight_project; additional templates can follow the same contract.

Execution back-plane

• agi-distributor API contains the Dask-based scheduler, worker templates and capacity balancer. Workers are packaged (python -m agi_node…) into ~/wenv/<app>_worker before run time.

• Optional cluster helpers (SSH, remote installs, zip staging) live under src/agilab/core/agi-node/agi_dispatcher and are reused by every app.

Runtime flow

1. A run configuration (Streamlit button, CLI script, PyCharm run config) calls an example in src/agilab/examples/<app>/AGI_run_*.py.

2. The script instantiates AgiEnv with the desired apps_dir and app. AgiEnv resolves symlinks, copies optional data bundles and loads app_settings.toml overrides.

3. AGI.run (or AGI.get_distrib / AGI.install) selects the dispatcher mode, builds or reuses the worker wheel, and starts a scheduler locally or on the configured SSH hosts.

4. agi-distributor API spins up workers, streams WorkDispatcher plans derived from the app manager, and feeds telemetry back into the capacity predictor.

5. Results land in ~/agi-space (for end users) or the repo data/export folders (for developers), while logs are mirrored to ~/log/execute/<app>/ for reproducibility.

Two common execution modes:

• Local notebook / laptop – scheduler + workers run on the same machine. Use this for prototyping and keep an eye on ~/log/execute/<app>/ for telemetry.

• Cluster / SSH hosts – scheduler runs locally, workers spawn remotely via the SSH helpers in agi_cluster.agi_distributor. Provide credentials via ~/.agilab/.env and rerun pycharm/setup_pycharm.py after editing run configurations so CLI wrappers stay synced.

Repository map

Generated by docs/gen-docs.py (gitignore-aware)

agilab/
__pycache__/ [excluded]
apps/
    __pycache__/ [excluded]
    builtin/
        flight_project/
            .pytest_cache/ [excluded]
            .venv/ [excluded]
            Modules/ [excluded]
            src/
                __pycache__/ [excluded]
                flight/
                    __pycache__/ [excluded]
                    __init__.py
                    flight.py
                    flight_args.py
                flight_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    flight_worker.c
                    flight_worker.py
                    flight_worker.pyx
            test/
                __pycache__/ [excluded]
            pyproject.toml
        mycode_project/
            .pytest_cache/ [excluded]
            .venv/ [excluded]
            Modules/ [excluded]
            src/
                __pycache__/ [excluded]
                mycode/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    mycode.py
                    mycode_args.py
                mycode_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    mycode_worker.py
                    mycode_worker.pyx
            test/
                __pycache__/ [excluded]
            pyproject.toml
    templates/
        dag_app_template/
            src/
                __pycache__/ [excluded]
                dag_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    dag_app.py
                    dag_app_args.py
                dag_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        fireducks_app_template/
            src/
                __pycache__/ [excluded]
                fireducks_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    fireducks_app.py
                    fireducks_app_args.py
                fireducks_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        pandas_app_template/
            src/
                __pycache__/ [excluded]
                pandas_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    pandas_app.py
                    pandas_app_args.py
                pandas_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        polars_app_template/
            src/
                __pycache__/ [excluded]
                polars_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    polars_app.py
                    polars_app_args.py
                polars_app_worker/
                    __pycache__/ [excluded]
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
    .gitignore
    __init__.py
    install.py
    README.md
apps-pages/
    __pycache__/ [excluded]
    view_autoencoder_latenspace/
        .venv/ [excluded]
        src/
            view_autoencoder_latentspace/
                __pycache__/ [excluded]
                __init__.py
                autoencoder_latentspace.py
                view_autoencoder_latentspace.py
        pyproject.toml
    view_barycentric/
        .venv/ [excluded]
        __pycache__/ [excluded]
        src/
            view_barycentric/
                __pycache__/ [excluded]
                __init__.py
                barycentric_graph.py
                view_barycentric.py
        __init__.py
        pyproject.toml
    view_maps/
        .venv/ [excluded]
        src/
            view_maps/
                __pycache__/ [excluded]
                __init__.py
                maps.py
                view_maps.py
        pyproject.toml
    view_maps_3d/
        .venv/ [excluded]
        src/
            view_maps_3d/
                __pycache__/ [excluded]
                __init__.py
                maps_3d.py
                view_maps_3d.py
        pyproject.toml
    view_maps_network/
        .venv/ [excluded]
        src/
            view_maps_network/
                __pycache__/ [excluded]
                maps_network_graph.py
                view_maps_network.py
        pyproject.toml
    .gitignore
    __init__.py
    README.md
core/
    __pycache__/ [excluded]
    agi-cluster/
        .venv/ [excluded]
        __pycache__/ [excluded]
        src/
            __pycache__/ [excluded]
            agi_cluster/
                agi_distributor/
                    __pycache__/ [excluded]
                    __init__.py
                    agi_distributor.py
                    cli.py
            __init__.py
        .gitignore
        __init__.py
        build.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        uv_config.toml
    agi-core/
        src/
            agi_core/
                __pycache__/ [excluded]
                __init__.py
        LICENSE
        pyproject.toml
        README.md
    agi-env/
        .pytest_cache/ [excluded]
        .venv/ [excluded]
        __pycache__/ [excluded]
        src/
            __pycache__/ [excluded]
            agi_env/
                __pycache__/ [excluded]
                resources/
                    .agilab/
                        .env
                        balancer_df.csv
                        balancer_model.pkl
                    mistral_offline/
                        data/
                            .gitignore
                            .gitkeep
                        README.md
                __init__.py
                agi_env.py
                agi_logger.py
                app_args.py
                defaults.py
                pagelib.py
                streamlit_args.py
            __init__.py
            pyproject.toml
        test/
            __pycache__/ [excluded]
            __init__.py
            clean_csv.py
            dummy_cmd.py
            test_agi_env.py
            test_app_args.py
            test_pagelib.py
            test_streamlit_args.py
        .gitignore
        __init__.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        uv_config.toml
    agi-node/
        .venv/ [excluded]
        __pycache__/ [excluded]
        src/
            __pycache__/ [excluded]
            agi_node/
                __pycache__/ [excluded]
                agi_dispatcher/
                    __pycache__/ [excluded]
                    __init__.py
                    agi_dispatcher.py
                    base_worker.py
                    build.py
                    post_install.py
                    pre_install.py
                dag_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    dag_worker.py
                fireducks_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    fireducks_worker.py
                pandas_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    pandas_worker.py
                polars_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    polars_worker.py
                __init__.py
                utils.py
            __init__.py
        __init__.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        SECURITY.md
        uv_config.toml
    test/
        __pycache__/ [excluded]
        __init__.py
        test_agi_core_smoke.py
        test_agi_distributor.py
        test_base_worker.py
        test_dag_worker.py
        test_fireducks_worker.py
        test_pandas_worker.py
        test_polars_worker.py
        test_work_dispatcher.py
    tools/
        __pycache__/ [excluded]
        __init__.py
        test_kill.py
    .gitignore
    __init__.py
    gen_app_script.py
    get_supported_python_versions.py
    install.ps1
    install.sh
examples/
    flight/
        __pycache__/ [excluded]
        AGI_command_flight.py
        AGI_install_flight.py
        AGI_run_flight.py
    mycode/
        __pycache__/ [excluded]
        AGI_install_mycode.py
        AGI_run_mycode.py
pages/
    __pycache__/ [excluded]
    ▶️ EDIT.py
    ▶️ EXECUTE.py
    ▶️ EXPERIMENT.py
    ▶️ EXPLORE.py
resources/
    __pycache__/ [excluded]
    help/
        __pycache__/ [excluded]
        __init__.py
        cluster-help.html
        edit_help.html
        execute_help.html
        experiment_help.html
        index.html
        roadmap.html
        views_help.html
    __init__.py
    agi_logo.png
    agilab_logo.png
    code_editor.scss
    config.toml
    custom_buttons.json
    info_bar.json
    theme.css
test/
    __pycache__/ [excluded]
    __init__.py
    test_model_returns_code.py
    test_python_versions.py
    test_streamlit.sh
.gitignore
__init__.py
agi_codex.py
AGILAB.py
install_apps.ps1
install_apps.sh
lab_run.py
LICENSE
MANIFEST.in
steps.toml
uv_config.toml

Regenerate the tree after file changes with uv run --group sphinx --dev docs/gen-docs.py.

Core vs optional apps

AGILAB (open-source framework) underpins optional apps that can be installed separately. Public docs only cover the open-source layers and built-in apps.

Documentation map

• Quick-Start – install/run instructions and links to sample commands.

• Introduction – background, motivation, and terminology.

• AGI Core Architecture – internals of the Streamlit/CLI façade.

• Framework API – reference for AGI.run and the dispatcher helpers.

See also

• Project Files Structure for details on each top-level folder.

• Framework API for the public AGI.* orchestration helpers.

• agi-env API for environment bootstrapping and dataset handling.