How-to Guides¶

Scope

Task recipes for External Secrets Operator (ESO) 2.x: install and upgrade, provider setup, syncing, templating, generators, PushSecret, multi-tenancy hardening, and troubleshooting. All manifests use the current APIs: external-secrets.io/v1 for ExternalSecret and stores, v1alpha1 for PushSecret and generators. Field defaults and policy values are in Reference; the reasoning behind them is in Explanation.

Commands & Recipes¶

The sections below are self-contained recipes. Replace names, regions, and hostnames with your own.

Installation¶

Install from the official chart repository. Pin the chart version so upgrades are deliberate.

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace \
  --version 2.11.0

# Verify: three Deployments (controller, webhook, cert-controller) should be Ready
kubectl get deploy -n external-secrets
kubectl get crds | grep external-secrets.io

To manage CRDs outside Helm (for example with a GitOps tool), install them with server-side apply, because the bundle exceeds the client-side apply annotation limit:

kubectl apply --server-side \
  -f https://raw.githubusercontent.com/external-secrets/external-secrets/v2.11.0/deploy/crds/bundle.yaml

helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace --version 2.11.0 --set installCRDs=false

Upgrade ESO Safely¶

Only the latest minor gets fixes, and upstream recommends upgrading one minor at a time (for example 2.9 -> 2.10 -> 2.11), reading each release note.

# See what is installed and what is available
helm list -n external-secrets
helm search repo external-secrets/external-secrets --versions | head

# Upgrade to the next minor only
helm upgrade external-secrets external-secrets/external-secrets \
  -n external-secrets --version 2.11.0 --reuse-values

Coming from v0.16 or older

Releases from v0.17.0 onward no longer serve external-secrets.io/v1beta1. Before upgrading past v0.16.x, change every manifest in Git from v1beta1 to v1 (the schema is otherwise the same), let v0.16.2 re-store the objects, then continue upgrading minor by minor.

# Find manifests still on v1beta1 in a GitOps repo
grep -rl "external-secrets.io/v1beta1" ./clusters

# Rewrite them (review the diff before committing)
grep -rl "external-secrets.io/v1beta1" ./clusters | xargs sed -i 's#external-secrets.io/v1beta1#external-secrets.io/v1#'

# Check which versions the API server has stored
kubectl get crd externalsecrets.external-secrets.io -o jsonpath='{.status.storedVersions}'

Run ESO in High Availability¶

Leader election is off by default. Turn it on whenever you run more than one controller replica, and raise concurrent for throughput.

# values-ha.yaml
replicaCount: 2
leaderElect: true
concurrent: 5
webhook:
  replicaCount: 2
podDisruptionBudget:
  enabled: true
  minAvailable: 1
serviceMonitor:
  enabled: true

helm upgrade external-secrets external-secrets/external-secrets \
  -n external-secrets --version 2.11.0 -f values-ha.yaml

Provider Setup¶

AWS Secrets Manager¶

Namespaced store that authenticates with IRSA through a ServiceAccount in the same namespace (the ServiceAccount carries the eks.amazonaws.com/role-arn annotation):

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: myapp
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

Cluster-wide variant; cluster-scoped stores must name the ServiceAccount's namespace:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

With EKS Pod Identity, associate the IAM role with the ESO controller's own ServiceAccount and omit auth entirely; ESO then uses the SDK default credential chain. serviceAccountRef cannot be combined with Pod Identity. Use service: ParameterStore for SSM Parameter Store.

HashiCorp Vault¶

Configure Vault's Kubernetes auth method and a read-only policy (run against your Vault; the host is your cluster's API endpoint):

vault auth enable kubernetes
vault write auth/kubernetes/config kubernetes_host="https://<kube-apiserver>:6443"

vault policy write myapp-read - <<'EOF'
path "secret/data/myapp/*" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/myapp \
  bound_service_account_names=myapp-eso \
  bound_service_account_namespaces=myapp \
  policies=myapp-read ttl=15m

Then point a SecretStore at it. path is the KV mount and version: v2 handles the data/ prefix, so remote keys are relative to the mount:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: myapp
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "myapp"
          serviceAccountRef:
            name: myapp-eso

For OpenBao, use the dedicated openbao provider (since 2.7) rather than the Vault provider.

GCP Secret Manager¶

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: gcp-secret-manager
  namespace: myapp
spec:
  provider:
    gcpsm:
      projectID: my-project
      auth:
        workloadIdentity:
          serviceAccountRef:
            name: eso-workload-id-sa
          clusterLocation: us-central1
          clusterName: my-gke-cluster

The referenced Kubernetes ServiceAccount must be bound to a GCP service account (or principal) with roles/secretmanager.secretAccessor.

ExternalSecret Definition¶

Sync individual properties of a JSON secret into a Kubernetes Secret, refreshed hourly:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: myapp
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: production/database
        property: password

The same pattern against the Vault store above, mapping two keys:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myapp-secret
  namespace: myapp
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: myapp-secret
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: myapp/db
        property: password
    - secretKey: API_KEY
      remoteRef:
        key: myapp/api
        property: key

Extract or Find Many Keys at Once¶

dataFrom.extract copies every property of one remote secret; dataFrom.find searches by name regex or tags. rewrite renames keys on the way in.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-config
  namespace: myapp
spec:
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: app-config
  dataFrom:
    - extract:
        key: production/app-config
    - find:
        name:
          regexp: "^production/feature-flags/.*"
      rewrite:
        - regexp:
            source: "production/feature-flags/(.*)"
            target: "FLAG_$1"

Build a TLS Secret with a Template¶

Render two provider properties into a typed kubernetes.io/tls Secret (template engine v2):

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: prod-tls
  namespace: myapp
spec:
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: prod-tls
    template:
      type: kubernetes.io/tls
      data:
        tls.crt: "{{ .tlsCert }}"
        tls.key: "{{ .tlsKey }}"
  data:
    - secretKey: tlsCert
      remoteRef:
        key: prod/cert
        property: certificate
    - secretKey: tlsKey
      remoteRef:
        key: prod/cert
        property: private_key

When the manifest is itself rendered by Helm, escape the braces (for example {{ "{{ .tlsCert }}" }} or backtick raw strings) so Helm does not evaluate them. Test templates offline with the esoctl template command from cmd/esoctl in the upstream repo.

Generate Credentials with a Generator¶

An ECR pull secret that refreshes before the 12-hour token expiry:

apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
metadata:
  name: ecr-gen
  namespace: myapp
spec:
  region: eu-west-1
  auth:
    jwt:
      serviceAccountRef:
        name: ecr-pull-sa
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: ecr-secret
  namespace: myapp
spec:
  refreshInterval: 1h
  target:
    name: ecr-secret
    template:
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: |
          {"auths": {"{{ .proxy_endpoint | replace "https://" "" }}": {"auth": "{{ printf "%s:%s" .username .password | b64enc }}"}}}
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: ECRAuthorizationToken
          name: ecr-gen

A random password that is generated once and never rotated by ESO (every refresh of a generator produces a new value, so use CreatedOnce):

apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
  name: db-password
  namespace: myapp
spec:
  length: 32
  digits: 5
  symbols: 5
  noUpper: false
  allowRepeat: true
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-bootstrap-password
  namespace: myapp
spec:
  refreshPolicy: CreatedOnce
  target:
    name: db-bootstrap-password
    creationPolicy: Orphan
    immutable: true
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: Password
          name: db-password

Push a Secret to a Provider with PushSecret¶

Copy a cert-manager-issued certificate into Vault; keep the Vault copy if the PushSecret is deleted:

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: push-app-tls
  namespace: myapp
spec:
  refreshInterval: 1h
  updatePolicy: Replace
  deletionPolicy: None
  secretStoreRefs:
    - name: vault-backend
      kind: SecretStore
  selector:
    secret:
      name: app-tls
  data:
    - match:
        secretKey: tls.crt
        remoteRef:
          remoteKey: myapp/tls
          property: certificate

The referenced store needs write permission in the provider (for Vault, a policy with create and update on the path). Check that the provider supports PushSecret in the provider table.

Distribute a Secret to Many Namespaces¶

ClusterExternalSecret stamps an ExternalSecret into every namespace whose labels match:

apiVersion: external-secrets.io/v1
kind: ClusterExternalSecret
metadata:
  name: registry-pull-secret
spec:
  externalSecretName: registry-pull-secret
  namespaceSelectors:
    - matchLabels:
        registry-access: "true"
  refreshTime: 1m
  externalSecretSpec:
    secretStoreRef:
      name: aws-secrets
      kind: ClusterSecretStore
    refreshInterval: 1h
    target:
      name: registry-pull-secret
    dataFrom:
      - extract:
          key: shared/registry

kubectl label namespace myapp registry-access=true
kubectl get clusterexternalsecret registry-pull-secret -o yaml | grep -A5 provisionedNamespaces

Restrict a ClusterSecretStore to Selected Namespaces¶

Without conditions, any namespace can read everything the store's credentials can read.

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: restricted-store
spec:
  conditions:
    - namespaces:
        - "team-a"
        - "team-b"
    - namespaceSelector:
        matchLabels:
          eso-tier: shared
    - namespaceRegexes:
        - "^team-c-.*$"
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-role"
          serviceAccountRef:
            name: eso-vault-sa
            namespace: external-secrets

Restrict Who Can Create Stores and PushSecrets¶

Give developers ExternalSecret rights in their namespace but only read access to stores and PushSecrets:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: external-secret-developer
  namespace: my-app
rules:
  - apiGroups: ["external-secrets.io"]
    resources: ["externalsecrets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["external-secrets.io"]
    resources: ["secretstores", "pushsecrets"]
    verbs: ["get", "list", "watch"]

Also disable kinds you do not use, both the CRD and its reconciler:

# values-hardened.yaml
processClusterPushSecret: false
processPushSecret: false
crds:
  createClusterPushSecret: false
  createPushSecret: false
rbac:
  serviceAccountTokenCreate: false
webhook:
  extraArgs:
    tls-ciphers: "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"

With rbac.serviceAccountTokenCreate: false, grant serviceaccounts/token create to the ESO ServiceAccount per target ServiceAccount with a Role that uses resourceNames.

Install ESO Namespace-Scoped¶

For a single-tenant install that cannot touch other namespaces (cluster kinds are implicitly disabled):

helm install external-secrets external-secrets/external-secrets \
  -n team-a --version 2.11.0 \
  --set scopedNamespace=team-a \
  --set scopedRBAC=true

Verify the ESO Image Signature¶

Images are signed keylessly with Cosign from the upstream release workflow:

cosign verify ghcr.io/external-secrets/external-secrets:v2.11.0 \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github.com/external-secrets/external-secrets/.github/workflows/release.yml@.*$'

Force a Refresh and Roll Workloads¶

ESO updates the Secret, but pods that read it as environment variables keep the old value until restarted.

# Trigger an immediate re-sync of one ExternalSecret
kubectl annotate externalsecret myapp-secret -n myapp force-sync=$(date +%s) --overwrite

# Restart consumers manually
kubectl rollout restart deployment/myapp -n myapp

To automate restarts, run a controller such as Stakater Reloader and annotate the Deployment with reloader.stakater.com/auto: "true". Volume-mounted Secrets are refreshed by the kubelet after its sync period, but the application must re-read the file.

Common Issues¶

Issue	Diagnosis	Fix
Secret not syncing	`kubectl describe externalsecret <name>` shows `SecretSyncedError`	Check the store status and the event message
Store not ready	`kubectl get secretstore,clustersecretstore -A` shows `Ready=False`	Fix provider endpoint, CA bundle, or credentials; flood gate skips dependent ExternalSecrets until fixed
Auth failure	ESO controller logs	Verify IRSA/Pod Identity/Workload Identity binding and the role or policy
`no matches for kind ... in version external-secrets.io/v1beta1`	Manifests still on `v1beta1`	Change `apiVersion` to `external-secrets.io/v1`
CRD apply fails with "metadata.annotations: Too long"	Client-side apply	Use `kubectl apply --server-side`
`SecretOwnedByOther`	Two ExternalSecrets target the same Secret with `Owner`	Give each a unique `target.name`, or use `Merge`
Refresh not working	`refreshPolicy` or `refreshInterval`	`refreshInterval: 0s` or `CreatedOnce`/`OnChange` never re-sync periodically; check `syncWindows` too
Provider throttling	`externalsecret_provider_api_calls_count` rising, 429 errors	Lengthen `refreshInterval`, use `dataFrom.extract` instead of many `data` entries
Warning about unmaintained provider	Admission warning or events	Expected for providers without a maintainer; annotate the store with `external-secrets.io/ignore-maintenance-checks: "true"` to silence events

Debugging¶

# Sync status across the cluster
kubectl get externalsecret -A
kubectl get pushsecret -A
kubectl describe externalsecret myapp-secret -n myapp

# Events for ESO objects
kubectl get events -n myapp --field-selector involvedObject.kind=ExternalSecret

# Controller logs (raise verbosity with --set log.level=debug)
kubectl logs -n external-secrets deploy/external-secrets --since=15m

# Webhook and cert-controller health
kubectl logs -n external-secrets deploy/external-secrets-webhook --since=15m
kubectl get validatingwebhookconfigurations | grep external-secrets