Feature: versioned pipeline stage templates

This note captures a recurring product problem observed in AGILab pipeline workflows: generated stages are still stored as Python snippets, even though the orchestrator knows much more structure than the saved text reveals.

AGILab previously relied on targeted snippet migrations to keep some older labs running. That approach has since been retired because it was too fragile and too opaque: a saved stage could change behaviour simply because the loader rewrote it. The current product direction is better, but still incomplete:

  • saved Python now remains exactly as written

  • stale generated snippets must be regenerated or re-imported explicitly

  • AGILab still lacks a structured, version-aware representation for generated pipeline stages

The proposed feature is to replace raw generated snippets with explicit, versioned pipeline stage templates whose identity belongs to the orchestration layer rather than to any single application.

Problem

Today, AGILab stores many generated stages as raw Python code in lab_stages.toml. Even when a stage originates from ORCHESTRATE, the saved artifact is still a snippet, not a structured execution spec.

This has several drawbacks:

  • the real source of truth is code text, not structured stage data

  • users cannot clearly see whether a stage is current, stale, or app-owned custom code

  • changes in app contracts still surface as snippet drift instead of schema drift

  • regeneration paths are explicit but not first-class in the saved data model

  • imports from orchestration and saved labs do not share a fully explicit lifecycle

Current product stance

This proposal does not suggest reintroducing silent snippet migration.

The intended behaviour remains:

  • no implicit Python rewrite when a lab is loaded

  • no hidden repair pass during execution

  • explicit regeneration or refresh whenever a generated snippet becomes stale

The missing piece is a better representation for generated stages, so AGILab can detect and explain drift without mutating saved Python behind the user’s back.

Proposal

Introduce two explicit stage kinds:

  • template for AGILab-generated stages

  • raw_python for fully custom snippets

For template stages, store a structured payload instead of Python as the source of truth:

  • template_id

  • template_version

  • app

  • action or task

  • engine

  • question

  • args

Python code becomes a rendered view or export artifact, not the canonical representation. This keeps execution inspectable while making the actual stage contract explicit and versionable.

The key design rule is that template_id must identify the structural shape owned by AGILab orchestration, not the business meaning of one particular app stage. In other words:

  • template_id says how the orchestrator should render, validate, and refresh the stage

  • app says which application is targeted

  • action or task says what that application should do

That avoids reintroducing app-specific coupling under a different name.

Example shape

[[pipeline]]
kind = "template"
template_id = "pipeline.agi_run.single_action"
template_version = 3
app = "example_project"
action = "reference_allocator"
engine = "agi.run"
question = "Compute reference allocations"

[pipeline.args]
data_in = "network_sim/pipeline"
data_out = "routing_reference/pipeline"
time_horizon = 16
trajectories_glob = "flight_trajectory/pipeline/*"
sat_trajectories_glob = "sat_trajectory/pipeline/Trajectory/*.csv"

In this model:

  • template_id is generic and reusable across apps

  • app and action carry the business-specific intent

  • changing one app action does not require inventing a new template family

Expected behaviour

When AGILab loads a template stage:

  • it compares the saved template_version with the current template registry

  • if the versions match, the stage renders and runs normally

  • if the versions differ, AGILab does not silently rewrite the stage

  • instead, AGILab marks the stage as outdated and offers an explicit refresh from template action

For raw_python stages:

  • AGILab stores and runs the code as-is

  • AGILab does not attempt implicit structural migration

  • the user remains responsible for keeping the snippet aligned with the runtime

Why this is better

  • removes fragile text-rewrite migrations from the critical path

  • makes stage drift visible instead of hidden

  • produces more readable lab_stages.toml diffs

  • reduces regressions caused by legacy path or contract rewrites

  • separates product-supported templates from user-owned custom code

  • keeps orchestration concerns separate from app concerns

Transition plan

Phase 1

Add the structured fields for newly created stages while keeping support for legacy code stages.

Phase 2

Replace ad-hoc legacy repair logic with explicit stale-stage detection for structured stages. Existing raw Python remains untouched.

Phase 3

Provide a one-shot converter for older labs:

  • convert known generated snippets into template stages

  • keep unknown snippets as raw_python

Phase 4

Retire the remaining compatibility helpers once the majority of active labs have been converted and the refresh workflow is well established.

Non-goals

  • banning custom Python snippets entirely

  • parsing arbitrary Python into structured stages

  • guaranteeing automated conversion for every hand-edited snippet

  • encoding app-specific meaning directly into template_id

Product impact

This feature improves trust in AGILab pipeline execution. Instead of forcing the user to reason about raw Python drift, AGILab would tell them exactly which stages are current, which are stale, and which are fully custom.

That is a cleaner contract for both end users and developers, and it reduces the maintenance burden of keeping generated stages aligned with changing application interfaces.

It also gives AGILab a clearer ownership boundary: orchestration owns template families and their versions, while applications only provide the runtime target and business action invoked by the stage.