Part 1

The shared layer: Agent Substrate

Both projects run their workloads as Actors on Agent Substrate. This page defines the Substrate terms you will see in both labs, shows where each project sits, and names the Substrate build each one needs.

After this page you can:

Define atespace, ActorTemplate, Actor, WorkerPool, Worker, and atenet.
Say which capabilities come from Substrate rather than from ax or kagent.
Name the Substrate build each project targets, and why the labs use one cluster each.

What Agent Substrate is

Agent Substrate is an agent execution runtime that maps a large set of Actors onto a smaller set of ready Workers, on the basis that agent-like applications are idle most of the time.¹² It manages each Actor's lifecycle (create, destroy, suspend, resume), assigns Actors to Workers in real time, and routes incoming traffic to them.¹² Substrate describes itself as a system for running agents at scale, not an SDK for building them.¹²

Both projects depend on it. The ax README says ax "runs on top of Agent Substrate for sandboxed execution", points to the upstream agent-substrate/substrate install guide, and expects the Substrate Control API at api.ate-system.svc.cluster.local:443.¹ The kagent README lists Substrate as the component that "runs the agents and manages their lifecycle, including suspension, resumption, and snapshots".⁸

They do not target the same build. ax's go.mod at v0.3.1 requires the upstream Substrate module.³³ kagent v1.0.0-alpha6 states "Kagent requires Substrate v0.3.0-alpha3" and replaces the upstream module with github.com/kagent-dev/substrate v0.3.0-alpha3, a fork that adds a Kubernetes credential provider, so the egress gateway can inject model API keys that the agent runtime never sees.³²³⁰ Both builds install into the ate-system namespace, so this tutorial's labs use one cluster each.

The building blocks

You will meet these Substrate terms in both labs. Each links to its glossary entry.

Atespace: The isolation boundary an Actor belongs to. An Actor is addressed by atespace and name, and atespaces are global records rather than Kubernetes namespaces.²⁰
ActorTemplate: The definition of an Actor class: container images and snapshot configuration. Creating one triggers a golden snapshot, and it is treated as immutable.²⁰
Actor: One instance derived from an ActorTemplate. It is the unit that is suspended and resumed, and it moves between Workers over its lifetime.²⁰
WorkerPool: A Kubernetes custom resource that declares warm compute capacity as a fleet of pre-started worker pods.²⁰
Worker: One worker pod in a WorkerPool. It hosts several Actors at once, each in its own sandbox.²⁰
atenet: The networking stack. Its router resumes suspended Actors on demand and routes traffic to the right worker pod.²⁰

Atespaces, Actors, and Workers live in Substrate's control-plane database because they change too often for etcd. Only WorkerPool and SandboxConfig are Kubernetes objects.²⁰

Where each project sits

Figure: ax and kagent are separate control planes that both call a Substrate Control API (ate-api) to run their workloads as Actors. In the labs, each talks to its own Substrate build on its own cluster.

Two boxes at the top: the ax control plane in the ax-system namespace (ax-server and Redis) and the kagent control plane in the kagent namespace (controller, A2A gateway, PostgreSQL). Each calls ate-api in the ate-system namespace. Agent Substrate below them provides atespaces, Actors, WorkerPools, snapshots, suspend and resume, guest services, and atenet networking, backed by its own PostgreSQL and object storage.

Sandboxing, suspend and resume, and snapshots are therefore largely Substrate features that each project exposes in its own way.¹²¹⁸ The differences live in the layer above: what you declare, where state is stored, and who owns the agent loop. The next page covers the loop.

Substrate itself brings stateful dependencies. Its kind quickstart installs "ate, PostgreSQL, rustfs", and snapshots go to object storage.¹² An ax install therefore runs Redis plus Substrate's PostgreSQL, and a kagent install runs its own PostgreSQL plus Substrate's.¹⁸¹²

kagent 0.x and Substrate

Comparisons written against kagent 0.x describe a different default architecture. In v0.10.2, the latest 0.x release, a default Agent runs as an ordinary Kubernetes Deployment.^v0.10.2

Substrate was not absent from 0.x, though. v0.10.2 already ships an optional SandboxAgent custom resource that "runs in an isolated sandbox on Agent Substrate".^v0.10.2 In 0.x Substrate is an opt-in path; in the 1.0 line every agent runs on it.⁸