Cooking Helm with the External Secrets Operator & Reloader

Do you know how to set environment variables at runtime? Of course, you know. It’s easy and there are multiple ways to do it:

export EXPORT=export
setenv EXPORT=export
EXPORT=export

If you need the easiest way to modify ENV:

env EXPORT=notexport

If you are looking for the easiest way to make it persistent in Kubernetes with Helm - use the External Secrets Operator - https://external-secrets.io/v0.8.1/

Firstly, we need a Helm template for the External Secrets CRD manifests (You don’t want to do it manually, do you?). That is why, you should implement two types of templates for your Helm chart:

• one for native Key/Value pairs;
• one for injecting files (in any format) to map them inside your Pod.

Preparing Helm templates

1. Template for regular Key/Values pairs

Let’s start with the first type of template - regular key/values pairs. As an example, we will pair it with the AWS Secrets Manager (we also need Terraform IAM Role and Policies too, but we can do that later). Let’s add a template code to our Helm chart:

chart/template/external-secrets.yaml

{{- if .Values.secret.enabled }}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: {{ include "chart.name" . }}
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-10"
    "helm.sh/resource-policy": delete
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: {{ include "chart.name" . }}
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: {{ .Release.Namespace }}/{{ include "chart.name" . }}
{{- end }}

Description of the manifest:

1. secretStoreRef - access to our secret store - in other words - AWS Secrets Manager backend;
2. Target - what we want to - Secret in Kubernetes;
3. Name of a secret in the AWS Secrets Manager.

2. Template for files

chart/template/deployment.yaml

{{- if ((.Values).secrets_format_yaml).enabled }}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: {{ include "chart.name" . }}-yaml
  labels:
    {{- include "chart.labels" . | nindent 4 }}
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    template:
      data:
        {{ .Values.secrets_format_yaml.file }}: |
          {{`{{ .secret }}`}}
  data:
  - secretKey: secret
    remoteRef:
      key: {{ .Release.Namespace }}/{{ include "chart.name" . }}-yaml
{{- end }}

As we can see, instead of a native Kubernetes Secret - we use a file in that Secret.

3. Linking secret and deployment for Key/Values pairs

chart/template/deployment.yaml

   {{- if .Values.secret.enabled }}
          envFrom:
          - secretRef:
              name: {{ include "chart.name" . }}
          {{- end }}

4. Linking secret and deployment for files mapping

chart/template/deployment.yaml

              {{- with .env.volumeMounts }}
              volumeMounts:
              {{- toYaml . | nindent 16 }}
              {{- end }}
          volumes:
          {{- if (($.Values).secrets_format_yaml).enabled }}
           - name: {{ .Values.secrets_format_yaml.name }}
             secret:
               defaultMode: 420
               secretName: {{ include "chart.name" . }}-yaml
          {{- end }}

5. Our helm values files

chart/values.yaml

secrets_format_yaml:
  enabled: true
  name: file2mount
  file: file.json

volumeMounts:
  - mountPath: /home/file.json
    name: file2mount

Installation

1. Installation of External Secrets

If you use ArgoCD for your cluster, here is the manifest for it:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 2
      backoff:
        duration: 2s
        factor: 2
        maxDuration: 1m
  destination:
    namespace: external-secrets
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: https://charts.external-secrets.io
    chart: external-secrets
    targetRevision: 0.7.0
    helm:
      values: |
        installCRDs: true
        webhook:
          port: 9443
        serviceAccount:
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::AWS_ID:role/externalsecrets

2. Adding Secret Store

Then, you need to connect your backend to External Secrets deployment. We can choose the ClusterSecretStore type cause it works cluster wide (I also deploy it with ArgoCD):

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
  namespace: external-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

And now, we have to create Terraform to connect our IAM role to Service Account with IRSA - https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

Reloader

If you want the pod to auto restart on secret changes in AWS Secrets Manager - install Reloader

You just need to add an annotation to your deployment:

chart/template/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "chart.name" . }}
  {{- with .Values.deploymentAnnotations }}
  annotations:
     {{- toYaml . | nindent 4 }}
   {{- end }}
  labels:
    {{- include "chart.labels" . | nindent 4 }}

chart/Values.yaml

deploymentAnnotations:
  reloader.stakater.com/auto: "true"

That’s all folks!

Go ahead and just deploy it!