AGILab Architecture

This page gives a single place to understand how the repository is organised, which services collaborate at runtime, and where to hook in when building a new app or extending the platform.

New to AGILab? Start with Quick-Start for a five-minute run-through, then return here when you need the big picture of how the layers fit together.

AGILab layers from the Streamlit front-end down to worker clusters.

Component view

Pyreverse snapshot of how the Streamlit entry points, agi_core façade, agi_env and agi_cluster exchange data before the workers are started.

Layers at a glance

Layer responsibilities

Layer / Code roots	What it does
User surfaces	Streamlit pages (src/agilab/pages) plus CLI mirrors in src/agilab/examples and tools/run_configs. They capture user intent and translate it into AGI.* calls.
agi_core	Reusable widgets, telemetry, app loaders. Keeps Streamlit/CLI code thin and prepares WorkDispatcher manifests. See AGI Core Architecture.
agi_env	Discovers datasets, resolves symlinks, stages bundles in ~/wenv and prepares environment variables. Every run builds an AgiEnv first.
agi_cluster / agi_node	Scheduler, workers, balancer, SSH/zip helpers. Turns manifests into Dask jobs locally or on remote hosts.
Apps (``src/agilab/apps``)	Project-specific logic. App managers describe build_distribution and worker arguments but rely on the shared layers above.
Workers (``~/wenv/<app>_worker``)	Cythonised/packaged code deployed on cluster nodes.

User surfaces

• Streamlit pages shipped in src/agilab/pages (EXECUTE/EXPLORE/EDIT/…).

• CLI mirrors under docs/source/apps-pages and tools/run_configs.

• Example scripts in src/agilab/examples (kept in sync via pycharm/setup_pycharm.py).

Core services

• agi-env API handles configuration, environment discovery, dataset staging and artifact caching. Every entry point constructs an AgiEnv before touching a worker.

• agi-node API and agi-cluster package the reusable logic shared between all apps (dataset helpers, worker bootstrap, git/LFS utilities…).

• Framework API exposes AGI.run / AGI.get_distrib / AGI.install orchestration.

• Apps under src/agilab/apps stay isolated but consume the same base worker / dispatcher APIs. Public apps live in mycode_project and flight_project; the rest use the same contract but ship privately.

Execution back-plane

• agi-cluster contains the Dask-based scheduler, worker templates and capacity balancer. Workers are packaged (python -m agi_node…) into ~/wenv/<app>_worker before run time.

• Optional cluster helpers (SSH, remote installs, zip staging) live under src/agilab/core/agi-node/agi_dispatcher and are reused by every app.

Runtime flow

1. A run configuration (Streamlit button, CLI script, PyCharm run config) calls an example in src/agilab/examples/<app>/AGI_run_*.py.

2. The script instantiates AgiEnv with the desired apps_dir and app. AgiEnv resolves symlinks, copies optional data bundles and loads app_settings.toml overrides.

3. AGI.run (or AGI.get_distrib / AGI.install) selects the dispatcher mode, builds or reuses the worker wheel, and starts a scheduler locally or on the configured SSH hosts.

4. agi-cluster spins up workers, streams WorkDispatcher plans derived from the app manager, and feeds telemetry back into the capacity predictor.

5. Results land in ~/agi-space (for end users) or the repo data/export folders (for developers), while logs are mirrored to ~/log/execute/<app>/ for reproducibility.

Two common execution modes:

• Local notebook / laptop – scheduler + workers run on the same machine. Use this for prototyping and keep an eye on ~/log/execute/<app>/ for telemetry.

• Cluster / SSH hosts – scheduler runs locally, workers spawn remotely via the SSH helpers in agi_cluster.agi_distributor. Provide credentials via ~/.agilab/.env and rerun pycharm/setup_pycharm.py after editing run configurations so CLI wrappers stay synced.

Repository map

Generated by docs/gen-docs.py (gitignore-aware)

agilab/
__pycache__/ [excluded]
apps/
    __pycache__/ [excluded]
    flight_clone_project/
        .pytest_cache/ [excluded]
        .venv/ [excluded]
        dataset_assets/
            beams.csv
            norad_3le.txt
            satellites.csv
        src/
            flight_clone/
                __pycache__/ [excluded]
                __init__.py
                app_args.py
                flight_clone.py
                flight_clone_args.py
            flight_clone_worker/
                __pycache__/ [excluded]
                __init__.py
                _satellite_helpers.py
                dataset.7z
                flight_clone_worker.c
                flight_clone_worker.py
                flight_clone_worker.pyx
                pyproject.toml
                sat_trajectory_worker.py
            app_args_form.py
            app_settings.toml
            flight_clone_args_form.py
            pre_prompt.json
        test/
            __pycache__/ [excluded]
            __init__.py
            _test_call_worker.py
            _test_flight_clone_manager.py
            _test_flight_clone_worker.py
            test_df.ipynb
            test_flight_clone_manager.py
            test_flight_clone_worker.py
        tools/
            example_waypoints.geojson
            generate_waypoints_basic.py
            generate_waypoints_complex.py
            synthetic_airports.geojson
        app_test.py
        distribution.json
        MANIFEST.in
        pyproject.toml
        README.md
        run_test.sh
        uv_config.toml
    flight_project/
        .pytest_cache/ [excluded]
        .venv/ [excluded]
        src/
            flight/
                __pycache__/ [excluded]
                __init__.py
                flight.py
                flight_args.py
            flight_worker/
                __pycache__/ [excluded]
                __init__.py
                dataset.7z
                flight_worker.py
                flight_worker.pyx
                pyproject.toml
            app_args_form.py
            app_settings.toml
            pre_prompt.json
        test/
            __pycache__/ [excluded]
            __init__.py
            _test_call_worker.py
            _test_flight_manager.py
            _test_flight_worker.py
            output.json
            test_df.ipynb
            test_flight_manager.py
            test_flight_worker.py
        distribution.json
        lab_steps.ipynb
        lab_steps.toml
        MANIFEST.in
        output.json
        pyproject.toml
        README.md
        run_test.sh
        steps.toml
        uv_config.toml
    mycode_project/
        .pytest_cache/ [excluded]
        .venv/ [excluded]
        src/
            mycode/
                __pycache__/ [excluded]
                __init__.py
                app_args.py
                mycode.py
                mycode_args.py
            mycode_worker/
                __pycache__/ [excluded]
                __init__.py
                mycode_worker.py
                mycode_worker.pyx
                pyproject.toml
            app_args_form.py
            app_settings.toml
        test/
            __pycache__/ [excluded]
            _test_call_worker.py
            _test_mycode_manager.py
            _test_mycode_worker.py
            test_mycode_manager.py
            test_mycode_worker.py
        pyproject.toml
        README.md
        uv_config.toml
        wenv
    network_sim_project/
        test/
            __init__.py
            test_network_sim_distribution.py
    templates/
        dag_app_template/
            src/
                dag_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    dag_app.py
                    dag_app_args.py
                dag_app_worker/
                    __init__.py
                    dag_app_worker.py
                    pyproject.toml
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        fireducks_app_template/
            src/
                fireducks_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    fireducks_app.py
                    fireducks_app_args.py
                fireducks_app_worker/
                    __init__.py
                    fireducks_app_worker.py
                    pyproject.toml
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        pandas_app_template/
            src/
                pandas_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    pandas_app.py
                    pandas_app_args.py
                pandas_app_worker/
                    __init__.py
                    pandas_app_worker.py
                    pyproject.toml
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
        polars_app_template/
            src/
                polars_app/
                    __pycache__/ [excluded]
                    __init__.py
                    app_args.py
                    polars_app.py
                    polars_app_args.py
                polars_app_worker/
                    __init__.py
                    polars_app_worker.py
                    pyproject.toml
                app_args_form.py
                app_settings.toml
                pre_prompt.json
            .gitignore
            MANIFEST.in
            pyproject.toml
            README.md
            uv_config.toml
    .gitignore
    __init__.py
    install.py
    README.md
apps-pages/
    view_autoencoder_latenspace/
        .venv/ [excluded]
        src/
            view_autoencoder_latentspace/
                __init__.py
                autoencoder_latentspace.py
                view_autoencoder_latentspace.py
        pyproject.toml
    view_barycentric/
        .venv/ [excluded]
        src/
            view_barycentric/
                __init__.py
                barycentric_graph.py
                view_barycentric.py
        __init__.py
        pyproject.toml
    view_maps/
        .venv/ [excluded]
        src/
            view_maps/
                __init__.py
                maps.py
                view_maps.py
        pyproject.toml
    view_maps_3d/
        .venv/ [excluded]
        src/
            view_maps_3d/
                __init__.py
                maps_3d.py
                view_maps_3d.py
        pyproject.toml
    view_maps_network/
        .venv/ [excluded]
        src/
            view_maps_network/
                maps_network_graph.py
                view_maps_network.py
        pyproject.toml
    .gitignore
    __init__.py
    README.md
core/
    __pycache__/ [excluded]
    agi-cluster/
        .venv/ [excluded]
        src/
            agi_cluster/
                agi_distributor/
                    __pycache__/ [excluded]
                    __init__.py
                    agi_distributor.py
                    cli.py
            __init__.py
        .gitignore
        __init__.py
        build.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        uv_config.toml
    agi-core/
        src/
            agi_core/
                __pycache__/ [excluded]
                __init__.py
        LICENSE
        pyproject.toml
        README.md
    agi-env/
        .pytest_cache/ [excluded]
        .venv/ [excluded]
        __pycache__/ [excluded]
        src/
            agi_env/
                __pycache__/ [excluded]
                resources/
                    .agilab/
                        .env
                        balancer_df.csv
                        balancer_model.pkl
                    mistral_offline/
                        data/
                            .gitignore
                            .gitkeep
                        README.md
                __init__.py
                agi_env.py
                agi_logger.py
                app_args.py
                defaults.py
                pagelib.py
                streamlit_args.py
            __init__.py
            pyproject.toml
        test/
            __pycache__/ [excluded]
            __init__.py
            clean_csv.py
            dummy_cmd.py
            test_agi_env.py
            test_app_args.py
            test_pagelib.py
            test_streamlit_args.py
        .gitignore
        __init__.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        uv_config.toml
    agi-node/
        .venv/ [excluded]
        src/
            agi_node/
                __pycache__/ [excluded]
                agi_dispatcher/
                    __pycache__/ [excluded]
                    __init__.py
                    agi_dispatcher.py
                    base_worker.py
                    build.py
                    post_install.py
                    pre_install.py
                dag_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    dag_worker.py
                fireducks_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    fireducks_worker.py
                pandas_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    pandas_worker.py
                polars_worker/
                    __pycache__/ [excluded]
                    __init__.py
                    polars_worker.py
                __init__.py
                utils.py
            __init__.py
        __init__.py
        LICENSE
        MANIFEST.in
        pyproject.toml
        README.md
        SECURITY.md
        uv_config.toml
    test/
        __pycache__/ [excluded]
        __init__.py
        test_agi_distributor.py
        test_base_worker.py
        test_dag_worker.py
        test_fireducks_worker.py
        test_pandas_worker.py
        test_polars_worker.py
        test_work_dispatcher.py
    tools/
        __init__.py
        test_dask.py
        test_kill.py
    .gitignore
    __init__.py
    gen_app_script.py
    get_supported_python_versions.py
    install.ps1
    install.sh
examples/
    flight/
        AGI_command_flight.py
        AGI_install_flight.py
        AGI_install_flight_legacy.py
        AGI_run_flight.py
        AGI_run_flight_legacy.py
    flight_clone/
        AGI_install_flight_clone.py
        AGI_run_flight_clone.py
    flight_legacy/
        AGI_install_flight_legacy.py
        AGI_run_flight_legacy.py
    flight_trajectory/
        AGI_install_flight_trajectory.py
        AGI_run_flight_trajectory.py
    ilp/
        AGI_install_ilp.py
        AGI_run_ilp.py
    link_sim/
        AGI_install_link_sim.py
        AGI_run_link_sim.py
    mycode/
        AGI_install_mycode.py
        AGI_run_mycode.py
    network_sim/
        AGI_install_network_sim.py
        AGI_run_network_sim.py
    sat_trajectory/
        AGI_get_distrib_sat_trajectory.py
        AGI_install_sat_trajectory.py
        AGI_run_sat_trajectory.py
    satcom_sim/
        AGI_install_satcom_sim.py
    sb3_trainer/
        AGI_install_sb3_trainer.py
        AGI_run_sb3_trainer.py
pages/
    __pycache__/ [excluded]
    ▶️ EDIT.py
    ▶️ EXECUTE.py
    ▶️ EXPERIMENT.py
    ▶️ EXPLORE.py
resources/
    help/
        __init__.py
        cluster-help.html
        edit_help.html
        execute_help.html
        experiment_help.html
        index.html
        roadmap.html
        views_help.html
    __init__.py
    agi_logo.png
    agilab_logo.png
    code_editor.scss
    config.toml
    custom_buttons.json
    info_bar.json
    theme.css
test/
    __pycache__/ [excluded]
    __init__.py
    test_model_returns_code.py
    test_python_versions.py
    test_streamlit.sh
.gitignore
__init__.py
agi_codex.py
AGILAB.py
install_apps.ps1
install_apps.sh
lab_run.py
LICENSE
MANIFEST.in
output.json
steps.toml
uv_config.toml

Regenerate the tree after file changes with uv run python docs/gen-docs.py.

Documentation map

• Quick-Start – install/run instructions and links to sample commands.

• Introduction – background, motivation, and terminology.

• AGI Core Architecture – internals of the Streamlit/CLI façade.

• Framework API – reference for AGI.run and the dispatcher helpers.

See also

• Project Files Structure for details on each top-level folder.

• Framework API for the public AGI.* orchestration helpers.

• agi-env API for environment bootstrapping and dataset handling.