Kubernetes¶

Install rivers on a Kubernetes cluster using Helm. For a tour of how the pieces interact, see Overview.

Before you begin¶

• A Kubernetes cluster (1.28+).
• kubectl configured against it.
• helm 3.8+ (OCI registry support).

Install with Helm¶

The CRDs are cluster-scoped and ship as a separate chart so multiple rivers namespaces can share them. Install them once per cluster, then the main rivers chart per namespace.

helm install rivers-crds \
  oci://ghcr.io/ion-elgreco/charts/rivers-crds \
  --version 0.6.3 \
  --namespace kube-system

helm install rivers \
  oci://ghcr.io/ion-elgreco/charts/rivers \
  --version 0.6.3 \
  --namespace rivers \
  --create-namespace

Wait for the operator and UI to come up:

kubectl -n rivers rollout status deploy/rivers-operator
kubectl -n rivers rollout status deploy/rivers-ui

Reach the UI¶

By default the UI service is ClusterIP. For local access:

kubectl -n rivers port-forward svc/rivers-ui 3000:3000
# open http://localhost:3000

For public exposure, set ui.serviceType=LoadBalancer (cloud) or front the service with your usual Ingress / Gateway.

Deploy a CodeLocation¶

A CodeLocation is a deployment unit — one image containing a Python module that exposes a CodeRepository. The operator pulls the image, resolves a tag to a digest, and runs rivers serve.

apiVersion: rivers.io/v1alpha1
kind: CodeLocation
metadata:
  name: analytics
  namespace: rivers
spec:
  image: ghcr.io/acme/pipelines
  tag: v0.2.0
  module: pipelines.analytics

kubectl apply -f analytics.yaml
kubectl -n rivers get codelocations
# NAME        PHASE   SOURCE                                  REPLICAS   AGE
# analytics   Ready   ghcr.io/acme/pipelines@sha256:abc1234   1/1        30s

Once PHASE=Ready the UI lists the location's assets, jobs, and run history. Materializations triggered from the UI dispatch back to the code-location's gRPC endpoint (spec.grpcPort, default 3001).

A fuller spec wires in resources, env, and secrets:

apiVersion: rivers.io/v1alpha1
kind: CodeLocation
metadata:
  name: analytics
  namespace: rivers
spec:
  image: ghcr.io/acme/pipelines
  tag: v0.2.0
  module: pipelines.analytics
  replicas: 1             # the maximum; code locations do not scale out yet
  digestRefreshInterval: 5m
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: "2"
      memory: 2Gi
  env:
    - name: SNOWFLAKE_ACCOUNT
      value: "acme-prod"
    - name: SNOWFLAKE_PASSWORD
      valueFrom:
        secretKeyRef:
          name: snowflake-creds
          key: password
  imagePullSecrets:
    - name: ghcr-pull-secret
  serviceAccountName: rivers-pipelines

Image rollouts¶

Runs get the last fully rolled-out image. When the tag moves to a new digest, or you change spec.tag or spec.digest, the operator rolls the code-location pods to the new image. status.resolvedImage and the SOURCE column move only when every pod runs the new image and is ready; the ImageResolved condition names the digest being rolled out. A pod is ready once its module has loaded and its gRPC server answers the readiness probe. Until then runs and the UI keep the previous digest, and the CodeLocation stays Ready on it.

While the rollout runs, DeploymentAvailable has reason RollingOut (ProgressDeadlineExceeded once it is stuck) and its message names the digest being rolled out and the one runs still use. If the new image's module does not load, rivers serve exits, and status.message and the end of the DeploymentAvailable message show code location <image>@sha256:... failed to load (Error, exit code 1): and the end of its log. Runs keep the previous digest until an image that loads rolls out. A first deployment stays Deploying until its first rollout completes.

kubectl get codelocation analytics -n old -o yaml \
  | yq '.metadata.namespace = "new"' \
  | kubectl apply -f -
kubectl delete codelocation analytics -n old

Git-sourced CodeLocations¶

A CodeLocation can take its code from a git repository instead of an image. The operator pins the ref to a commit, the pods fetch that commit and install its dependencies with uv, and every run uses the commit of the pod that launched it.

apiVersion: rivers.io/v1alpha1
kind: CodeLocation
metadata:
  name: analytics
  namespace: rivers
spec:
  git:
    url: https://github.com/acme/pipelines.git
    ref:
      branch: main            # exactly one of branch | tag | commit
    path: analytics           # subdirectory holding the project
    secretRef:
      name: git-creds
  module: analytics.pipeline

kubectl -n rivers get rcl
# NAME        PHASE   SOURCE         REPLICAS   AGE
# analytics   Ready   main@9f3c1ab   1/1        45s

Git-sourced CodeLocations covers refs, dependencies, credentials, the runtime image, the shared workspace and troubleshooting. The workspace reference covers storage, retention and outages.

Helm chart customizations¶

Drop these into a values.yaml and pass -f values.yaml on helm install / upgrade:

operator:
  replicas: 2
  # In production, leave false. Enable only for in-cluster HTTP-only
  # registries (e.g. k3d's local registry); CodeLocations against an
  # HTTP registry must otherwise pre-set spec.digest.
  allowInsecureRegistry: false
  git:
    # In production, leave false. http:// git urls send the code and the
    # git credentials unencrypted; enable only for a git host on a trusted
    # network.
    allowInsecure: false
    # Time for one ref resolution, in whole seconds above 0; the operator
    # does not start with any other value.
    timeoutSeconds: 30

  webhook:
    # selfSigned (default): chart generates a self-signed CA + serving
    #   cert at install time. No external dependency.
    # certManager: emit a cert-manager Certificate; requires a working
    #   cert-manager and an issuerRef.
    certProvider: certManager
    certManager:
      issuerRef:
        name: letsencrypt-prod
        kind: ClusterIssuer

ui:
  enabled: true
  # ClusterIP (default), NodePort, or LoadBalancer
  serviceType: ClusterIP

surrealdb:
  enabled: true
  persistence:
    size: 10Gi

The full set of values is documented in deploy/helm/rivers/values.yaml.

Authenticated SurrealDB connections¶

The bundled SurrealDB always runs authenticated. Every rivers pod (operator, UI, code-location, run, step) signs in as a database-scoped user (DEFINE USER ... ON DATABASE) on every connection — credentials flow via valueFrom.secretKeyRef so the password value never lands in a pod spec or CR.

With the bundled SurrealDB (default)¶

Out of the box, helm install creates three Secrets and a Job without any extra values:

• rivers-surrealdb-bootstrap — root creds for the bundled DB pod. Username defaults to rivers-root, password is randAlphaNum 32 on first install and preserved across helm upgrade via Helm's lookup.
• rivers-surrealdb-auth — database-scoped rivers user. Username defaults to rivers, password auto-generated and preserved the same way. This is the Secret every rivers pod mounts.
• rivers-surrealdb-setup — pre-rendered SurrealQL file with the rivers user definition (password baked in from the same helper as the auth Secret). Mounted by the user-init Job.
• <release>-surrealdb-user-init-r<revision> Job: applies the setup file via surreal import against the bundled DB using the bootstrap root creds. DEFINE USER ... OVERWRITE makes the auth Secret the source of truth — rotate the Secret then helm upgrade to rotate the in-DB user. The Job runs as a regular resource (not a Helm hook); a wait-for-surreal init container on operator/UI pods gates them on SurrealDB readiness so they don't crashloop while the bundled pod is still starting.

To set the rivers user creds explicitly (instead of auto-generating):

surrealdb:
  auth:
    username: rivers-prod
    password: change-me   # dev convenience; for prod use `existingSecret` instead

To bring your own Secret (production path):

kubectl -n rivers create secret generic rivers-prod-creds \
  --from-literal=username=rivers-prod \
  --from-literal=password='...'

surrealdb:
  auth:
    existingSecret: rivers-prod-creds
    secretKeys:
      username: username
      password: password

The user-init Job then defines that user inside the bundled DB.

With external SurrealDB¶

Define the user yourself and point the chart at the Secret:

-- Once, against your external SurrealDB:
DEFINE NAMESPACE IF NOT EXISTS rivers;
USE NS rivers;
DEFINE DATABASE IF NOT EXISTS main;
USE NS rivers DB main;
DEFINE USER `rivers-prod` ON DATABASE PASSWORD '...' ROLES OWNER;

kubectl -n rivers create secret generic rivers-surrealdb \
  --from-literal=username=rivers-prod \
  --from-literal=password='...'

surrealdb:
  enabled: false
  endpoint: wss://surreal.example.com:443
  auth:
    namespace: rivers
    database: main
    existingSecret: rivers-surrealdb

No bootstrap Secret or init Job is created — those only exist for the bundled DB. Rotate the user externally and update the Secret; the next helm upgrade (or pod restart) picks up the new value via the secretKeyRef mount.

External SurrealDB without auth¶

For an unauthenticated external SurrealDB (e.g. a closed dev cluster), just omit auth.existingSecret and auth.username/password:

surrealdb:
  enabled: false
  endpoint: ws://surreal.dev:8000
  # auth.existingSecret / auth.username / auth.password all empty → no signin

Pods connect without signin, matching surreal start --unauthenticated.

Why the bundled DB is always authenticated¶

There's no useful "unauthenticated bundled DB" mode — once SurrealDB is sharing a cluster with rivers, anything else in the namespace can dial ws://surrealdb:8000 and read/write the orchestration state. The chart removes the footgun by always requiring auth for the bundled path.

Local dev (rivers dev)¶

rivers dev reads the same env vars (RIVERS_SURREAL_USERNAME / RIVERS_SURREAL_PASSWORD / RIVERS_SURREAL_NAMESPACE / RIVERS_SURREAL_DATABASE) for --surreal-endpoint connections, or none of them when using embedded storage. Set them in your shell when pointing rivers dev at an authenticated remote SurrealDB.

UI authentication¶

The UI ships unauthenticated (ui.auth.mode: none). Before exposing it via Ingress or HTTPRoute, enable one of the two auth modes:

ui:
  auth:
    mode: oidc                        # or "forward" behind an auth proxy
    publicUrl: https://rivers.example.com
    oidc:
      issuer: https://keycloak.example.com/realms/main
      clientId: rivers
      existingSecret: rivers-oidc-client   # key: client-secret

oidc speaks OpenID Connect (code flow + PKCE) directly to your IdP; forward trusts identity headers injected by an authenticating reverse proxy (Authelia, oauth2-proxy, Envoy Gateway, …) from an explicit trustedProxies CIDR list. Invalid combinations fail helm install loudly. See the authentication guide for the full option set, proxy header mappings, allowlists, and the launched-by audit trail.

OpenTelemetry export¶

Every pod that runs rivers Python code (code-location, run, step) can export traces over OTLP/gRPC. The operator and UI do not export traces. Set the endpoint once; the operator stamps it on every pod it creates, and the run pod stamps it on its step pods. The chart gives the endpoint to the operator as RIVERS_OTEL_ENDPOINT, so an OTEL_EXPORTER_OTLP_ENDPOINT that something else injects into the operator pod does not reach your pods:

otel:
  endpoint: https://otlp.example.com:4317

Most hosted backends need an API key or bearer token. Put it in a Secret in OTLP header form and reference the Secret. The value reaches pods via valueFrom.secretKeyRef and never lands in a pod spec or CR:

kubectl -n rivers create secret generic otel-headers \
  --from-literal=headers='authorization=Bearer ...'

otel:
  endpoint: https://otlp.example.com:4317
  headers:
    existingSecret: otel-headers   # key: headers

https:// endpoints use TLS with the image's system roots plus bundled Mozilla roots, so images without a CA bundle work too. To override the endpoint or headers for one code location, set the same OTEL_EXPORTER_OTLP_* variables in CodeLocation.spec.env; each entry there replaces the chart value of the same name. A node-local collector endpoint such as http://$(HOST_IP):4317 cannot go in otel.endpoint. Leave otel.endpoint unset and put two entries in CodeLocation.spec.env: first HOST_IP from a status.hostIP fieldRef, then the endpoint. Kubernetes expands $(HOST_IP) only from entries earlier in the list.

Spans from every pod have the service name rivers. To tell code locations apart in your backend, set OTEL_SERVICE_NAME in each CodeLocation.spec.env, for example analytics. Its run and step pods get the same name. The environment variable reference lists every variable rivers reads.

Open ports¶

Upgrade¶

The chart, operator/UI images, and CRDs all release on the same vX.Y.Z tag. To upgrade in place:

helm upgrade rivers-crds \
  oci://ghcr.io/ion-elgreco/charts/rivers-crds \
  --version 0.2.0 -n kube-system

helm upgrade rivers \
  oci://ghcr.io/ion-elgreco/charts/rivers \
  --version 0.2.0 -n rivers

Existing CodeLocation resources are re-reconciled against the new operator without re-creation.

Then migrate the storage schema once, before the upgraded UI and code locations open the database — a newer build refuses an older database until it is migrated. Run it from any image that has the new rivers:

rivers db migrate --surreal-endpoint ws://surrealdb:8000

A migration can also raise the oldest build allowed to write (see Schema versioning). When it does, rebuild every CodeLocation image on the new rivers version right after migrating: an image built on an older rivers is refused as a writer and its code location stops serving. Schema versions 5 and 6 (asset actions) both raise the write floor.

Uninstall¶

kubectl delete codelocations -n rivers --all  # operator cleans up child resources first
helm uninstall rivers -n rivers
helm uninstall rivers-crds -n kube-system     # only when no other namespaces use rivers