Part 3

Setup: shared prerequisites

One checklist for both labs. The two projects need different builds of Agent Substrate, so this page sets up the parts they share, installs upstream Substrate v0.3.0 for the ax lab, and leaves the kagent build of Substrate to the kagent lab.

After this page you can:

  • State which Substrate build each project needs, and why they cannot share one install.
  • Check that a cluster serves the certificates beta APIs Substrate depends on.
  • Install upstream Substrate v0.3.0 on kind and prove that it is up.

Before you start

Work through this list once; the lab pages then list only what is specific to each project.

  1. A supported cluster. Substrate v0.3.0 documents two install paths: a kind quickstart for development and a GKE quickstart.12 Its GCP tooling states that Kubernetes versions below 1.36 are not supported on GKE;13 that is a GKE guide limit, not a universal minimum, because kagent's own kind setup runs Kubernetes 1.35.0 with the required APIs switched on.34 No EKS, AKS, or generic-cluster path is documented.12
  2. The certificates beta APIs. Substrate requires certificates.k8s.io/v1beta1/podcertificaterequests and certificates.k8s.io/v1beta1/clustertrustbundles (see below).13
  3. The right Substrate build for each lab. ax targets upstream Agent Substrate: its README sends you to the agent-substrate/substrate install guide, and its go.mod at v0.3.1 requires the upstream module.133 kagent v1.0.0-alpha6 does not. Its docs state "Kagent requires Substrate v0.3.0-alpha3", and its go.mod replaces the upstream module with the fork github.com/kagent-dev/substrate v0.3.0-alpha3, which adds the credential provider that injects model API keys at egress.3230 Both builds install into ate-system, so use one cluster per lab. This page installs upstream v0.3.0 for the ax lab; the kagent lab creates its own cluster with the fork.
  4. Labelled worker nodes (ax lab). On upstream Substrate, worker capacity only schedules on nodes that carry the ate.dev/substrate-version label. The installer labels the nodes that exist when it runs; nodes added later, for example by an autoscaler, need the label applied by hand.12 kagent's fork chart and kind scripts neither set nor use this label.31
  5. Client tools. Docker, kind, and kubectl for both labs; Go, ko, and a registry your cluster can pull from for ax;1 Helm, jq, and openssl for kagent, because its Substrate and kagent installs drive Helm charts and mint certificates.31
  6. A model API key for the provider you plan to use (Anthropic or Google for ax, OpenAI by default for kagent).427 Each lab stores it in a Kubernetes Secret read from standard input, so the key never appears in a command line.

Install upstream Substrate on kind (ax lab)

The Substrate README's development quickstart is the shortest documented path. Run it from a checkout of agent-substrate/substrate at tag v0.3.0, with Go, kubectl, and Docker installed.12

git clone --branch v0.3.0 https://github.com/agent-substrate/substrate.git
cd substrate
hack/create-kind-cluster.sh
hack/install-ate-kind.sh --deploy-ate-system

The first script creates a kind cluster and a local registry, and enables the certificates beta APIs on that cluster. The second installs the Substrate control plane, PostgreSQL, and rustfs (an S3-compatible object store).1213 This install has no credential provider configured, so it does not serve the kagent lab.32

Verify Substrate

ax looks for the Substrate Control API Service named api in ate-system, and its README gives this check.1 The same two commands also verify the kagent lab's Substrate once it is installed.

kubectl get svc api -n ate-system
kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version

Expected result: the first command lists one Service named api. The second lists the atelet DaemonSet with a SUBSTRATE-VERSION column showing the installed version.112 On the kagent lab's cluster that column is empty, because the fork does not set the label; that is expected there.

If it does not work: a NotFound error for service "api" means Substrate is not installed in ate-system, or kubectl points at another cluster; check kubectl config current-context, then rerun the install script and read its output before continuing.

The certificates beta API requirement

Substrate components use short-lived Pod certificates as their mutual TLS identity20, so Substrate v0.3.0 needs two beta APIs: certificates.k8s.io/v1beta1/podcertificaterequests and certificates.k8s.io/v1beta1/clustertrustbundles.13 On GKE they are supported on 1.36 only if enabled at cluster creation, or on GKE 1.37 and later.13

In Kubernetes 1.36 these APIs are beta and off by default; they become stable and on by default in 1.37.14 Substrate v0.3.0 calls the v1beta1 client specifically, so a cluster that serves only the v1 API would not satisfy it either.13 The real requirement is therefore that your cluster serves the certificates.k8s.io/v1beta1 PodCertificateRequest and ClusterTrustBundle APIs, not a particular version number. kagent's own kind setup shows this: it runs Kubernetes 1.35.0 and turns on the same APIs through the ClusterTrustBundle, ClusterTrustBundleProjection, and PodCertificateRequest feature gates and the certificates.k8s.io/v1beta1: "true" runtime config.34 On managed clusters that cannot enable beta APIs or feature gates, this remains a blocker.