Thoughts on the Kubernetes Secret Store CSI Driver with AWS

Last year, I evaluated using the secrets store CSI driver for Kubernetes with the AWS provider.

It’s terrible. Don’t use it.

I needed a way to use secrets stored in AWS Secrets Manager in EKS.

Our engineers were used to having secrets available to their services via environment variables. Like this:

apiVersion: v1
kind: Pod
metadata:
  name: env-single-secret
spec:
  containers:
  - name: envars-test-container
    image: nginx
    env:
    - name: SECRET_USERNAME
      valueFrom:
        secretKeyRef:
          name: backend-user
          key: backend-username

I wanted to keep this interface so we don’t have to modify any of the services’ code.

The CSI driver wants you to make a SecretProviderClass. Then you can refer to the SecretProviderClass in your Pod definition by mounting a volume like this:

# Vault not AWS Secrets Manager example pulled from the docs
kind: Pod
apiVersion: v1
metadata:
  name: secrets-store-inline
spec:
  containers:
    - name: busybox
      image: registry.k8s.io/e2e-test-images/busybox:1.29
      command:
      - "/bin/sleep"
      - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-sync"

You can sync secrets as Kubernetes secrets instead of doing the volume like normal. But the configuration is utterly confusing.

The following is pulled straight from the AWS provider for the driver:

Consider a secret “MySecret” with JSON content as follows:

    {
        "username": "testuser",
        "password": "testpassword"
    }

To mount the username and password key pairs of this secret as individual secrets, use the jmesPath field as follows:

      objects: |
          - objectName: "MySecret"
            objectType: "secretsmanager"
            jmesPath:
                - path: "username"
                  objectAlias: "MySecretUsername"
                - path: "password"
                  objectAlias: "MySecretPassword"          

Listen, it took me hours to figure out how to map some key-value pairs to a SecretProviderClass.

The field names sound so similar and are confusing.

I still barely know what I’m doing. There’s no way I’d be able to tell non-infrastructure engineers to read the docs and figure it out.

What should you use instead? Probably external-secrets. I haven’t used it yet, but that’s the direction we’re going to move in.