Skip to main content

Temporal Worker Controller

The Temporal Worker Controller provides automation to enable rainbow deployments of your Workers by simplifying the tracking of which versions still have active Workflows, managing the lifecycle of versioned Worker deployments, and calling Temporal APIs to update the routing config of Temporal Worker Deployments. The Temporal Worker Controller makes it simple and safe to deploy Temporal Workers on Kubernetes.

If you run versioned Workers on Kubernetes, the Worker Controller is the recommended way to manage rollouts and autoscaling together.

Why adopt the Worker Controller?​

The traditional approach to revising Temporal Workflows is to add branches using the Versioning APIs. Over time these checks can become a source of technical debt, as safely removing them from a codebase is a careful process that often involves querying all running Workflows.

Worker Versioning is a Temporal feature that allows you to pin Workflows to individual versions of your Workers, which are called Worker Deployment Versions. Using pinning, you will not need to add branching to your Workflows to avoid non-determinism errors. This allows you to bypass the other Versioning APIs.

The Worker Controller gives you direct, programmatic control over your Worker deployments, and integrates with the Temporal CLI. You do not need to use the Worker Controller to use Worker Versioning, but when used together, Worker Versioning and the Worker Controller can provide more graceful deployments and upgrades, and less need to manually tune your Workers.

Note that in Temporal, Worker Deployment is sometimes referred to as Deployment, but since the Worker Controller makes significant references to Kubernetes Deployment resource, within this page we will stick to these terms:

  • Worker Deployment: A Worker Deployment is a logical service that groups similar Workers together for unified management. Each Deployment has a name (such as your service name) and supports versioning through a series of Worker Deployment Versions.
  • Worker Deployment Version: A Worker Deployment Version represents an iteration of a Worker Deployment. Each Deployment Version consists of Workers that share the same code build and environment. When a Worker starts polling for Workflow and Activity Tasks, it reports its Deployment Version to the Temporal Server.
  • Deployment: A Kubernetes Deployment resource. A Deployment is "versioned" if it is running versioned Temporal workers/pollers.

Features​

  • Registration of new Temporal Worker Deployment Versions
  • Creation of versioned Deployment resources (that manage the Pods that run your Temporal pollers)
  • Deletion of resources associated with drained Worker Deployment Versions
  • Manual, AllAtOnce, and Progressive rollouts of new versions
  • Ability to specify a "gate" Workflow that must succeed on the new version before routing real traffic to that version
  • Autoscaling of versioned Deployments using Kubernetes Horizontal Pod Autoscaler (HPA)

Refer to the Temporal Worker Controller repo for usage details.

Autoscaling versioned Workers​

The Worker Controller can manage autoscaling for versioned worker Deployments without forcing you to choose between safe rollout behavior and elastic capacity.

Use the Worker Controller when you need all of the following:

  • Worker Versioning for safe Workflow code changes
  • Kubernetes-native rollout automation
  • autoscaling that follows each active Worker Deployment Version separately

Because the Worker Controller uses Kubernetes HPA, you can scale on any metric available to your HPA pipeline, including:

  • CPU and memory utilization
  • Task Queue backlog metrics exposed through your metrics pipeline
  • slot utilization and other Worker-specific metrics
  • custom metrics surfaced through Prometheus or another Kubernetes metrics adapter

TemporalWorkerOwnedResource​

To attach autoscaling or other Kubernetes resources to each Worker Deployment Version, use a TemporalWorkerOwnedResource (TWOR).

A TWOR lets you define a resource template once and have the Worker Controller create a version-specific copy for each active Worker Deployment Version. This is useful for resources such as:

  • HorizontalPodAutoscaler
  • PodDisruptionBudget
  • other Kubernetes resources that should track the lifecycle of a versioned Deployment

The Worker Controller manages these resources alongside the versioned Deployments it creates, so they are updated and cleaned up as versions roll forward and drain.

Why use this instead of KEDA?​

If you are already using the Worker Controller for Worker Versioning, prefer the Worker Controller for autoscaling as well. This keeps rollout management and scaling attached to the same versioned Kubernetes Deployments.

KEDA can still be a valid option for non-versioned or legacy worker deployments. However, for versioned Workers, the Worker Controller is the preferred path because it keeps autoscaling aligned with Worker Deployment Versions.

Configuring Worker Lifecycles​

To use the Temporal Worker Controller, tag your Workers following the guidance for using Worker Versioning.

Here is an example of a progressive rollout strategy gated on the success of the HelloWorld Workflow:

rollout:
  strategy: Progressive
  steps:
    - rampPercentage: 1
      pauseDuration: 30s
    - rampPercentage: 10
      pauseDuration: 1m
  gate:
    workflowType: "HelloWorld"

As you ship new deployment versions, the Worker Controller automatically detects them and gradually makes that version the new Current Version of the Worker deployment it is a part of. As older pinned Workflows finish executing and deprecated deployment versions become drained, the Worker Controller also frees up resources by sunsetting the Deployment resources polling those versions.

When you use autoscaling with the Worker Controller, each active Worker Deployment Version can scale independently while it is serving traffic. This allows older versions to drain safely while newer versions scale based on live demand.

Running the Temporal Worker Controller​

You can install the Temporal Worker Controller using our Helm chart:

RELEASE=temporal-worker-controller
NAMESPACE=temporal-system
VERSION=1.0.0

helm install $RELEASE oci://docker.io/temporalio/helm-charts/temporal-worker-controller \
  --version $VERSION \
  --namespace $NAMESPACE \
  --create-namespace
  
helm install temporal-worker-controller ./helm/temporal-worker-controller \
  --namespace $NAMESPACE \
  --create-namespace

Refer to GitHub for other Worker Controller deployment templates.