Arcade Deploy on your own cluster (Helm)
This page is for platform operators who run Arcade with the Helm chart and want developers to ship custom servers into that installation with arcade deploy. Once you turn deployments on, the runs each deployed server as a workload in your own cluster. Bundles go to an object store you own, and runner images come from a registry you choose.
Bundles stay in your environment: the chart requires an artifact store (deployments.artifacts.url), and the publishes bundles only there. Usage-event tracking (usageTracking.enabled) is off by default in the chart. Leave it off so your installation sends no usage events to Arcade.
The deployments.* values first shipped in chart 1.11.0. Use chart 1.11.1 or later: in 1.11.0 you must set deployments.image.repository yourself, or the chart fails to render, and there are no deployments.runtime.* values.
How it works
- A developer runs
arcade deployagainst your installation. The publishes the server’s bundle to your artifact store (S3-compatible, GCS, or Azure Blob). - The creates the server in the workers namespace (
arcade-workersby default). Each deployed server runs as a Kubernetes Deployment of the runner image (arcadedev/runner), one pod by default, and the pod fetches its bundle from the artifact store. - Every deployed server gets its own NetworkPolicy. It accepts traffic only from the . Outbound, it reaches the public internet and cluster DNS, but not private address ranges, link-local addresses (including cloud metadata), or the cluster ranges you list, unless you allowlist a range.
The ’s service gets a namespaced Role in the workers namespace. That Role covers Deployments, Services, Secrets, ServiceAccounts, NetworkPolicies, ResourceQuotas, Leases, read access to pods and pod logs, and read access to pod metrics. Developers never need cluster access. See Engine service account permissions for the full list and the bucket permissions.
Prerequisites
- A working Arcade installation from the Helm chart, with
gateway.hostname,ingress.hostname, ordeployments.domainset - A bucket in an S3-compatible store, Google Cloud Storage, or Azure Blob Storage for release bundles
- Credentials for that bucket that the can pick up from its environment. The Arcade Engine reads them from the ambient cloud credential chain, never from the bucket URL and never from a deploy request. See Bucket credentials.
- Access from your cluster to the runner image, either
arcadedev/runneron Docker Hub or a copy in your own registry - A CNI plugin that enforces Kubernetes NetworkPolicies, so the per-server isolation takes effect
Bucket credentials
The opens the artifact store with each cloud SDK’s default credential lookup. Give credentials to the Arcade Engine’s ServiceAccount, <release>-engine in the release namespace, or put them in its environment. Pick one method:
| Method | Store | Chart |
|---|---|---|
| EKS Pod Identity | S3 | 1.11.1 or later |
| Static keys in a Secret | S3, Azure Blob | 1.11.1 or later |
| EKS IRSA | S3 | 1.11.2 or later |
| Azure Workload Identity | Azure Blob | 1.11.2 or later |
| GKE Workload Identity | GCS | 1.11.2 or later |
| GCP service account key file | GCS | 1.11.2 or later |
The identity you give the needs to read and write objects in the bucket: it writes each bundle, and the download URL it signs is only valid if that identity can read the object. See Engine service account permissions for the exact permissions. Deployed servers don’t need bucket credentials as long as the Arcade Engine can sign download URLs. See Artifact store.
EKS Pod Identity
EKS Pod Identity needs no chart values. With the EKS Pod Identity Agent installed, create an IAM role that EKS Pod Identity can assume and associate it with the ’s ServiceAccount. For a release named arcade in the arcade namespace:
aws eks create-pod-identity-association \
--cluster-name my-cluster \
--namespace arcade \
--service-account arcade-engine \
--role-arn arn:aws:iam::111122223333:role/arcade-engineSee EKS Pod Identity for the role’s trust policy.
Static keys
Put the keys in a Secret in the release namespace and pass it to the with engine.extraEnvFrom:
- S3:
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY - Azure Blob:
AZURE_STORAGE_ACCOUNTandAZURE_STORAGE_KEY. A shared key also lets the sign download URLs.
kubectl -n arcade create secret generic arcade-bundle-store \
--from-literal=AWS_ACCESS_KEY_ID=... \
--from-literal=AWS_SECRET_ACCESS_KEY=...engine:
extraEnvFrom:
- secretRef:
name: arcade-bundle-storeWorkload identity and key files
Workload identity isn’t available before chart 1.11.2.
Set up the identity on the cloud side first, following EKS IRSA , Azure Workload Identity , or GKE Workload Identity for the ’s ServiceAccount. Then set the matching values:
| Value | Default | Description |
|---|---|---|
engine.serviceAccount.annotations | {} | Annotations on the Arcade Engine’s ServiceAccount. |
engine.podLabels | {} | Labels on the Arcade Engine’s pods. Keys the chart already sets are rejected. |
engine.podAnnotations | {} | Annotations on the Arcade Engine’s pods. checksum/* keys are rejected. |
engine.extraVolumes | [] | Extra volumes for the Arcade Engine’s pods. |
engine.extraVolumeMounts | [] | Extra volume mounts for the Arcade Engine’s container. |
The identity applies to the ’s Deployment only. The Arcade Engine’s Jobs (migration, gateway bootstrap, worker reconcile) use their own ServiceAccounts. Extra volumes and mounts also go into those Jobs.
EKS IRSA:
engine:
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/arcade-engineAzure Workload Identity. Also name the storage , with AZURE_STORAGE_ACCOUNT in engine.extraEnv or storage_account= in deployments.artifacts.url:
engine:
serviceAccount:
annotations:
azure.workload.identity/client-id: 00000000-0000-0000-0000-000000000000
podLabels:
azure.workload.identity/use: "true"GKE Workload Identity:
engine:
serviceAccount:
annotations:
iam.gke.io/gcp-service-account: arcade-engine@my-project.iam.gserviceaccount.comGCP service key file, mounted from a Secret named gcp-key with a key.json entry:
engine:
extraVolumes:
- name: gcp-key
secret:
secretName: gcp-key
extraVolumeMounts:
- name: gcp-key
mountPath: /var/secrets/gcp
readOnly: true
extraEnv:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/secrets/gcp/key.jsonEngine service account permissions
The ’s ServiceAccount, <release>-engine, needs permissions in two places: the workers namespace in your cluster, and the artifact store bucket.
Kubernetes
The chart creates all of this when deployments.enabled is true. You don’t need to add anything. It renders a Role named <release>-engine-workloads in the workers namespace and a RoleBinding with the same name that binds it to the <release>-engine ServiceAccount in the release namespace. The Role applies to one namespace only: the chart creates no ClusterRole, and the gets no permissions outside the workers namespace.
| API group | Resources | Verbs |
|---|---|---|
apps | deployments | get, list, watch, create, update, patch, delete |
core ("") | services, secrets, serviceaccounts | get, list, watch, create, update, patch, delete |
networking.k8s.io | networkpolicies | get, list, watch, create, update, patch, delete |
coordination.k8s.io | leases | get, list, watch, create, update, patch |
core ("") | resourcequotas | get, list, watch, create, update, patch |
core ("") | pods | get, list, watch |
core ("") | pods/log | get |
metrics.k8s.io | pods | get, list |
With deployments.createNamespace: false, the chart doesn’t create the workers namespace, but it still renders the Role and RoleBinding into it. Create the namespace before you install or upgrade.
Bucket
The performs two operations on the bucket:
- Writes each bundle. On
arcade deploy, it uploads the bundle as one object at a key built from the tenant, , server name, version, and the bundle’s SHA-256. Deploying the same bundle again can write to the same key, so a write can replace an existing object. - Signs a download URL for each bundle. The requests a 24-hour expiry, and temporary credentials can shorten it. The deployed server’s pod uses the URL to fetch its bundle. See Artifact store.
The doesn’t list or delete objects, and it doesn’t read objects itself. It doesn’t remove bundles from the bucket, so apply any retention you want with a bucket lifecycle rule. A signed URL only works if the identity that signed it can read the object, so the identity needs read access in practice. Read access lets a pod download its bundle.
The table lists the permissions each cloud requires for those two calls.
| Store | Write a bundle | Signed download URL |
|---|---|---|
| S3 | s3:PutObject. Bundles of 16 MiB or more upload in parts, which s3:PutObject also covers. Add s3:AbortMultipartUpload so a failed upload cleans up. | s3:GetObject. Any credential that can sign works, including STS session credentials. |
| GCS | storage.objects.create. Replacing an existing object also needs storage.objects.delete. | storage.objects.get. Without a key file, the service account also needs iam.serviceAccounts.signBlob on itself. |
| Azure Blob | Role Storage Blob Data Contributor, or a shared key. | A shared key (AZURE_STORAGE_KEY). Workload and managed identities can’t sign. |
For GCS, the predefined role roles/storage.objectUser on the bucket includes storage.objects.create, storage.objects.get, and storage.objects.delete. Grant roles/iam.serviceAccountTokenCreator on the service itself to cover iam.serviceAccounts.signBlob. For more on GCS permissions, see IAM permissions for Cloud Storage .
For S3, this policy covers both operations, scoped to one bucket. Replace arcade-bundles with your bucket name:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:AbortMultipartUpload"
],
"Resource": "arn:aws:s3:::arcade-bundles/*"
}
]
}Enable deployments
Add the deployments values
Deployments are off by default. Turn them on and configure your artifact store. These are the only values you must add. See Minimal working configuration:
deployments:
enabled: true
artifacts:
# s3://, gs:// or azblob://
url: s3://arcade-bundles?region=us-east-1Required schemes are one of:
s3://gs://azblob://
Upgrade the release
Apply the new values to your existing release:
helm upgrade arcade \
oci://public.ecr.aws/s5i6x9d1/charts/arcade \
--namespace arcade \
-f values.yamlWith the defaults, this creates the arcade-workers namespace, a default-deny NetworkPolicy for it, a ResourceQuota and LimitRange, and the ’s Role in that namespace. To use a namespace you manage yourself, see Workers namespace (deployments.createNamespace: false).
Deploy a test server
Before the first deploy, make sure the developer has:
- An in your installation, and a in it they can deploy to.
- Signed in with
arcade login --url https://arcade.example.com, and selected that witharcade project set <project-id>if it isn’t the default.
Then ask them to deploy a server and check that it reaches Running. If the deploy fails, check the logs and the pod events in the workers namespace.
Minimal working configuration
This is the only addition a working Arcade install needs to turn deployments on. The rest of the install comes from the Helm chart guide. The hostname must already be set through gateway.hostname, ingress.hostname, or deployments.domain, and the needs credentials for the bucket as described in Bucket credentials.
deployments:
enabled: true
artifacts:
# S3. For Google Cloud Storage use gs://arcade-bundles,
# for Azure Blob Storage use azblob://arcade-bundles
url: s3://arcade-bundles?region=us-east-1Everything under deployments other than these two values keeps its default. The hostname comes from gateway.hostname or ingress.hostname, so you don’t need deployments.domain.
Configuration
All settings live under deployments.* in the chart values. Leave a value at its default unless you need to change it.
Core settings
| Value | Default | Description |
|---|---|---|
deployments.enabled | false | Turns deployments on. |
deployments.provider | kubernetes | The deployment provider. kubernetes runs deployed servers in your cluster. |
deployments.mutationsAllowed | true | When false, the Arcade Engine blocks creating and updating deployments. |
deployments.domain | "" | Root domain for deployments, passed to the Arcade Engine. Falls back to gateway.hostname, then ingress.hostname, and the chart refuses to render if none is set. |
deployments.timeout | 5m | How long a deploy may take to become healthy before it’s marked failed. |
deployments.autoRoll | true | When true, the reconciler moves deployed servers onto a new runner image one at a time (see Upgrades). When false, the move is deferred and deployed servers stay on the runner image they already run until they’re redeployed. A redeploy picks up the configured runner image. |
deployments.reconciler.enabled | true | Runs the deployment reconciler. It moves deployed servers onto a new runner image and removes NetworkPolicies left behind by removed servers. |
deployments.readinessProbe | rollout | How the Arcade Engine decides a deployed server is ready. rollout waits for the Kubernetes rollout to finish, which needs the pod to accept TCP connections on the server’s port. Any other value, such as health, also waits until the server’s /worker/health endpoint answers HTTP 200 to the Arcade Engine. |
Artifact store
| Value | Default | Description |
|---|---|---|
deployments.artifacts.url | "" | Bucket URL for release bundles, for example s3://bundles?region=us-east-1, gs://bundles, or azblob://bundles. Required when deployments are on. |
Deployed servers’ pods get no bucket credentials: their ServiceAccount mounts no token, and their NetworkPolicy blocks the cloud metadata endpoint. Instead, the gives each pod a presigned download URL for its bundle when it can sign one:
| Store | Signs when | Otherwise |
|---|---|---|
| S3 | Any credential. With STS session credentials, the URL expires with the session. | The pod falls back to the store URL, which it has no credentials for. |
| GCS | A key file is configured, or the Arcade Engine’s service account holds iam.serviceAccounts.signBlob. | The pod falls back to the store URL, which it has no credentials for. |
| Azure Blob | A shared key is configured (AZURE_STORAGE_KEY). | Workload and managed identities can’t sign, so the pod falls back to the store URL, which it has no credentials for. |
Arcade will fail to pull the server bundle if the signed URL is invalid. With Azure Workload Identity or GKE Workload Identity, make sure signing works as described above.
Runner image and registry
| Value | Default | Description |
|---|---|---|
deployments.image.repository | arcadedev/runner | Runner image that deployed servers run in. |
deployments.image.tag | "" | Runner image tag. Empty uses the chart’s appVersion. |
deployments.registry.server | "" | Registry host to pull the runner image from. When set, it’s prefixed to deployments.image.repository. |
deployments.registry.imagePullSecrets | [] | Names of image pull Secrets for deployed servers’ pods. When dockerRegistry.enabled is true and the chart creates that Secret, the chart adds it to this list. |
For a private mirror, push arcadedev/runner to your registry and point the chart at it:
deployments:
registry:
server: registry.example.internal
imagePullSecrets:
- my-registry-pull-secretSizing
Every deployed server gets the same size.
| Value | Default | Description |
|---|---|---|
deployments.sizing.instances | 1 | Instances per deployed server. Keep it at 1: Arcade Deploy runs one serving instance per server, without high availability or autoscaling. A higher value makes the Arcade Engine run that many pods per server. |
deployments.sizing.cpuLimit | 1.0 | CPU limit per deployed server, in cores. |
deployments.sizing.memoryLimit | 512 | Memory limit per deployed server, in mebibytes (an integer, not a quantity string). |
deployments.runtime.className | "" | RuntimeClass for deployed servers, for example a sandboxed runtime. Empty uses the cluster default. |
deployments.runtime.nodeSelector | {} | Node selector for deployed servers’ pods. |
deployments.runtime.tolerations | [] | Tolerations for deployed servers’ pods. |
With the chart, each pod’s requests equal its limits, so pods run in the Guaranteed QoS class. Each pod also has a fixed ephemeral-storage request and limit of 6Gi, which covers a 5Gi volume for the bundle and its virtual environment.
Workers namespace
| Value | Default | Description |
|---|---|---|
deployments.namespace | arcade-workers | Namespace deployed servers run in. |
deployments.createNamespace | true | Creates the namespace with the restricted Pod Security Standard (enforce, audit, and warn) and a default-deny NetworkPolicy. Helm keeps both on uninstall. |
deployments.resourceQuota.enabled | true | Renders a ResourceQuota named <release>-workers-quota from deployments.resourceQuota.hard. |
deployments.resourceQuota.hard | requests.cpu: 10, requests.memory: 20Gi, limits.cpu: 20, limits.memory: 40Gi, pods: 50 | Hard limits for that quota. |
deployments.limitRange.enabled | true | Renders a LimitRange for containers in the namespace. |
deployments.limitRange.default | cpu: 1, memory: 512Mi | Default container limits. |
deployments.limitRange.defaultRequest | cpu: 250m, memory: 256Mi | Default container requests. |
To use a namespace you manage yourself, set deployments.createNamespace: false. The chart then adds no default-deny NetworkPolicy, unless you also set deployments.network.defaultDenyExistingNamespace: true. Only do that when the namespace holds nothing but deployed servers.
When the reconciler is on, the also applies its own ResourceQuota, arcade-deployments-budget, in the workers namespace. It limits pods, limits.cpu, and limits.memory, and the reconciler re-applies it on every pass, so manual edits to it don’t last. A pod has to fit every quota in the namespace.
Size deployments.resourceQuota.hard for the number of servers you expect, at deployments.sizing.cpuLimit and deployments.sizing.memoryLimit each, and make sure your nodes can hold them. If deploys stop creating pods while the chart quota still has room, check arcade-deployments-budget with kubectl -n arcade-workers get resourcequota.
If the chart already rendered its quota, only a helm upgrade with deployments.resourceQuota.enabled: false removes it. The ’s Role can’t delete ResourceQuotas.
Network
| Value | Default | Description |
|---|---|---|
deployments.egressAllowlist | [] | CIDRs deployed servers may reach in addition to the public internet, for example an internal API. |
deployments.network.clusterCIDRs | [] | Your cluster’s pod and service CIDRs. Deployed servers can’t reach them. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, and 169.254.0.0/16 stay unreachable as well. Ranges in deployments.egressAllowlist are reachable, and so is DNS (TCP and UDP port 53) to the resolvers below. |
deployments.network.dns.mode | cluster | cluster uses cluster DNS. external replaces pod DNS with deployments.network.dns.nameservers and removes every cluster-DNS exception. |
deployments.network.dns.nameservers | [] | Public IPv4 nameservers. Required in external mode. |
deployments.network.dns.namespace | kube-system | Namespace of the cluster DNS pods (cluster mode only). |
deployments.network.dns.podSelector | {} | Labels of the cluster DNS pods, for example k8s-app: kube-dns (cluster mode only). Empty allows every pod in that namespace. |
deployments.network.dns.cidrs | [] | Extra resolver IP ranges, such as a NodeLocal DNS /32. These allow TCP and UDP port 53 only (cluster mode only). |
external DNS mode also needs a bundle endpoint the pod can reach without cluster DNS. Existing deployed servers pick up a DNS mode change only after a redeploy.
Deploy a server
Developers use the same CLI as on Arcade Cloud. Each developer needs an and a in your installation first. They sign in to your installation once, select the project if needed, then deploy from the project directory:
arcade login --url https://arcade.example.com
arcade deploy -e src/my_server/server.pySee Arcade Deploy for signing in, picking a and deploying, and the CLI cheat sheet for every flag.
- Deploy servers built with the Arcade MCP framework for Python. The refuses a bundle without a Python definition or with a missing entrypoint.
- The refuses a changed redeploy while the previous one is still rolling out.
- The also refuses a bundle over its 16 MiB limit or one that unpacks past 256 MiB, and discards a virtual environment that the bundle ships. The 16 MiB limit caps the upload request, with a little headroom for encoding. Both limits are built into the Arcade Engine; no chart or configuration value changes them.
- A redeploy that changes the server’s code or secrets replaces the release under the same name and gateway URL. A redeploy of a running release that changes nothing is a no-op. Deploying again after a failed release rolls it out again, even if nothing changed.
For non-interactive CI deploys, set ARCADE_URL and ARCADE_API_KEY (a ) in the CI environment, then run arcade deploy. With both set, the CLI skips the saved and browser sign-in and uses ARCADE_URL directly as the ’s URL, without discovery. This needs arcade-mcp 1.16.0 or later.
Check status and logs
Dashboard and CLI
The Servers page in the dashboard lists deployed servers. A removed server disappears from the list. The tracks each deployed server in one of these states, and arcade deploy waits for a final one:
| State | Meaning |
|---|---|
pending | A new server is rolling out. The dashboard shows Deploying. |
updating | A redeploy is rolling out. The Arcade Engine refuses another redeploy of the server until it finishes. The dashboard shows Updating. |
running | The current release is ready and serving. |
degraded | No instance of the current release is serving, for example because it keeps crashing, or a rollout failed while the previous instance still serves. The status message gives the reason. |
failed | The release didn’t become ready, because it failed to start or deployments.timeout ran out, and nothing is serving it. |
From the CLI:
arcade server list
arcade server get <name>
arcade server logs <name> -fLogs cover the current release. After a crash, they also include the immediately previous instance. Arcade doesn’t keep logs beyond that, so ship pod logs to your own logging stack if you need them longer.
kubectl
Deployed servers are ordinary Kubernetes objects in the workers namespace, so your existing tooling works:
kubectl -n arcade-workers get deployments,pods,services
kubectl -n arcade-workers get networkpolicies -l app.kubernetes.io/managed-by=arcade-engine
kubectl -n arcade-workers get resourcequotaUpgrades
Upgrade the chart with helm upgrade as described in Self-host with Helm. Keep these points in mind:
- Runner image. With
deployments.image.tagempty, the runner tag follows the chart’sappVersion. Pindeployments.image.tagto control which runner tag the configures. - Private registry. Mirror the new runner tag into your registry before you upgrade.
- Namespace. With
deployments.createNamespace: true, Helm keeps the workers namespace and its default-deny NetworkPolicy on uninstall, so deployed servers’ objects aren’t deleted with the release.
Moving deployed servers onto a new runner image
When an upgrade changes the runner image tag, the reconciler moves running deployed servers onto it one at a time. This needs deployments.reconciler.enabled: true.
- On each pass (every 30 seconds), the reconciler starts at most one move, and only when no other move is still rolling out.
- If a server can’t start on the new runner image, the rolls it back to the previous one and marks it
degradedwith the message “could not start on the upgraded runtime; the previous runtime is still serving”. If no instance of the server is serving when it’s markeddegraded, the reason is “release artifact unavailable” instead. The move then halts: no other server moves onto that image. - Set
deployments.autoRoll: falseto defer the move. Deployed servers then stay on the runner image they already run until they’re redeployed.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
helm upgrade fails with deployments.artifacts.url must name an S3-compatible artifact store when deployments.enabled is true | Set deployments.artifacts.url. |
helm upgrade fails with deployments.domain (or gateway.hostname / ingress.hostname) must be set when deployments.enabled is true | Set deployments.domain, gateway.hostname, or ingress.hostname. |
helm upgrade fails with engine.extraConfig sets deployments: | The chart owns the deployments: block in the Arcade Engine’s configuration. Use deployments.* chart values instead of engine.extraConfig. |
| The Arcade Engine doesn’t start and names an unsupported scheme | Use an s3://, gs://, or azblob:// URL in deployments.artifacts.url. |
| A deploy fails because the server never became healthy | The pod started but didn’t answer its health check within deployments.timeout. Read the server’s logs, and check that it binds the port the platform probes. If many servers fail at once, suspect the runner image. If start-up is just slow, raise deployments.timeout. |
| Deploys stop creating pods | A ResourceQuota in the workers namespace is full. Run kubectl -n arcade-workers get resourcequota to see which one. For <release>-workers-quota, raise deployments.resourceQuota.hard and make sure your nodes can hold the extra servers. If arcade-deployments-budget is full, the Arcade Engine’s own cap is reached; remove deployed servers you no longer need. |
| A deployed server can’t reach an internal service | Add the service’s CIDR to deployments.egressAllowlist. |
| NetworkPolicies stay behind after you remove servers | With deployments.reconciler.enabled: false, nothing removes them. Delete NetworkPolicies labeled app.kubernetes.io/managed-by=arcade-engine by hand, or turn the reconciler back on. |
Next steps
- Deploy an MCP server with Arcade Deploy
- Create an MCP Gateway to give clients access to deployed servers’
- Platform architecture