Last updated on September 18th, 2026 at 04:03 pm
Most Kubernetes introductions show how to create a Secret item in just three lines of YAML. They never reveal that their secrets, by default, are base64-encoded strings in etcd, that aren’t encrypted, that aren’t rotated or audited.
It’s no small mistake. That’s a security hole that has been exploited in real life.
Base64 is encoding and NOT encryption. It only takes a moment, and anyone with etcd access can crack your database password. But if RBAC isn’t managed closely in clusters, that’s a much greater number of individuals than you might think.
Real secrets management starts at the architecture level—not at the kubectl create secret step.
Table of Contents
What “Secrets Management in Kubernetes” Actually Means
It is good to be specific about the problem space before starting work on the tools.
Kubernetes secrets management addresses 3 different concerns:
- Storage Security: Do secrets get encrypted at rest in etcd?
- Access control: Who/what may read a secret, and when?
- Lifecycle management: How are secrets rotated, versioned, and revoked?
Most teams solve 1 of these. Mature setups can address all three. The difference between the two states is most often where incidents occur.
Secrets Management in Kubernetes: External KMS, Vaults, and Encryption – The Real Architecture Breakdown
Encryption at Rest with KMS Providers
Kubernetes supports envelope encryption for storing secrets in etcd. The DEK (Secret) encrypts the secret; the KEK encrypts the DEK. The KEK exists outside the cluster and is stored in a Key Management Service, such as AWS KMS, Google Cloud KMS, or Azure Key Vault.
Even if someone drops your etcd, you only get encrypted blobs. Otherwise, they are worthless blobs.
Creating this will involve creating an EncryptionConfiguration file on the API server itself:
apiVersion: apiserver.config.k8s.io/v1kind: EncryptionConfigurationresources: - resources: - secrets providers: - kms: name: myKMSPlugin endpoint: unix:///tmp/socketfile.sock apiVersion: v2 - identity: {}The final identity provider is a “last resort” and a footgun. Without KMS, Kubernetes stores secrets unencrypted. I see teams miss this step, set it up, and forget about it, which makes it a silent Time Capsule winner.
This printer was observed in a staging environment, where the KMS plugin’s connectivity failed, and secrets were being delivered in a plain old fashion. The team saw no warnings and no errors. Worth auditing explicitly.
HashiCorp Vault – More Than Just a Secret Store
For reasons that shall remain a mystery, Vault is the most widely used external secrets backend in the Kubernetes world, and for good reason. Supports out-of-the-box dynamic secret generation, short-lived credentials, fine-grained policies, and full audit logging.
The team typically uses the Vault Agent Sidecar Injector pattern. Vault deploys an init container and sidecar to your pods, which authenticate via Kubernetes service accounts and extract secrets to a shared in-memory volume.
You shouldn’t have any secrets in the environment variables no secrets in Secret objects in Kubernetes. No data is stored; the pod gets what it needs at runtime.
In this, it excels:
- Database credentials – rotating hourly.
- Dynamic certificates for each of the services provided. Vault generates dynamic certificates for each required service.
- Assigning cloud provider credentials with narrowly-drafted IAM policies
When things get tricky:
- Operational investment in the HA setup at Vault is vital.
- When Vault is sealed and the agent can’t authenticate, the cold start will take a long time.
- Don’t mess around at 2 AM if you have a problem with sidecar injection; it’s not fun, okay?
While sidecar has been the recommended approach, I have tried this in a few different cluster configurations and consistently get good results using Vault Kubernetes auth in tandem with the External Secrets Operator (ESO). It provides more granular control over secret syncing and supports drift detection.
External Secrets Operator – The Bridge That Makes Sense
ESO is an open-source operator that securely accesses external secret stores (like Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and so on) and transforms the contents into native Kubernetes Secret objects.
The process is as follows:
- You create a SecretStore or ClusterSecretStore that references your external SecretStore.
- You define an ExternalSecret resource which correlates specific keys in the external store to a Kubernetes Secret.
- ESO fetches, syncs, and optionally rotates the secret every N minutes.
apiVersion: external-secrets.io/v1beta1kind: ExternalSecretmetadata: name: db-credentialsspec: refreshInterval: 1h secretStoreRef: name: vault-backend kind: SecretStore target: name: db-secret data: - secretKey: DB_PASSWORD remoteRef: key: secret/data/prod/db property: password
What you get is a standard Kubernetes Secret that ESO owns, and automatically refreshed at ESO’s mercy. ESO detects changes to the underlying value in Vault and updates the Secret without involving users.
It means as much as it does as a sound bite. When teams only rotate the hands out in panic mode, it is their own fault. The possibility of human error is eliminated by automating it.
For teams committed to securing Kubernetes, ESO is fast becoming the bare minimum—not an add-on—the Environment Variable Trap.
Many articles skim over this; sometimes, even after making Vault and ESO work, they end up in environment variables, and that is a bad thing.
The environment variables that are available in a container are:
- Available to any process in the container
- Found by certain frameworks on crashes or debugging
- If isolation between the container and the host isn’t perfect, the host can access the/proc/environ file.
The preferred alternative is to have secret files in an in-memory tmpfs. The data is not stored on disk, and the application reads it from the filesystem path.
volumes: - name: secrets-vol emptyDir: medium: MemoryvolumeMounts: - name: secrets-vol mountPath: /var/secrets readOnly: true
Though it is a minor pod spec change, this has a dramatic impact on the security posture – particularly in environments where Kubernetes Network Security policies are not consistently blocking intra-pod traffic.
My Take on Cloud-Native KMS Options
AWS Secrets Manager + ASCP
AWS Secrets and Configuration Provider (ASCP) provides the AWS Secrets Store CSI Driver. It mounts secrets directly as volumes via the CSI interface, with authentication using IRSA (IAM Roles for Service Accounts).
If you’re already on AWS, the integration is tight. The drawback: vendor lock-in – too late to move to GCP or multi-cloud later on and have to re-imagine your secret fetching layer.
GCP Secret Manager
Very much like Google’s offering, but less polluted by its IAM model. The External Secrets Operator supports Workload Identity well. The External Secrets Operator has good support for Workload Identity (WI).
Unlike AWS, my experience shows that GCP’s audit logging for secret access is also more granular out of the box. Every attempt to access a secret appears in the Cloud Audit Logs without setting anything up.
Azure Key Vault
While CSI’s implementation for Azure is working now, it has had a few bumps in the road with identifying pod identity. Things improved significantly with the transition from AAD Pod Identity to Azure AD Workload Identity (via Workload Identity). However, it still wasn’t as seamless as it could be without a migration effort on existing clusters.
What Most People Misunderstand About Secret Rotation
But the world of rotation seems simple – change the number, update the references, woop dee doo, that’s it! In practice, it’s a distributed coordination problem.
Once a secret rotates, if no running pod understands it, that pod begins to fail. The naive solution is to restart all pods after a rotation. What is more correct is to create applications that can deal with secret reload without rebooting – that is, reading it every time it is requested, or every time it’s refreshed after a fixed period.
Some frameworks include built-in support for this. Others require a container. Consider this before building the rotation pipeline.
It is important that you know two patterns:
- Blue/Green secret rotation – Duplicating the new secret on the old one, gradually moving the consumers in each of those versions over to the new one, and finally removing the old one. When using on-behalf-of, you can map database passwords to a particular secret version in your config with versioned secret references. When rotation occurs, it becomes an explicit, auditable deployment change.
Audit Logging – The Part Everyone Skips Until They Need It
Log access to secrets. This is not “a pod read a secret”; this is which service account, which pod, at what time, with what specific key.
When you enable/configure the right verbosity for Kubernetes API audit logs, you can see secret read events too. Vault has a native audit backend that does just this. Both AWS and GCP expose it in their respective logging systems.
It is the teams that do not do this step that have to scramble around after an incident, asking the other teams, “How long did the attacker have access?
Creating a simple audit policy for Kubernetes:
apiVersion: audit.k8s.io/v1kind: Policyrules: - level: Metadata resources: - group: "" resources: ["secrets"]
Stores metadata parameters (Who, When, What resource, but not secret value!) It’s a good place to start without polluting your audit log with sensitive data.
Putting It Together – A Practical Stack That Works
What follows is a realistic configuration for production that is secure, complex enough for operations, and maintainable:
| Secret Store | HashiCorp Vault (or AWS/GCP equivalent) | Dynamic secrets, fine-grained policies, audit logs |
| Sync Layer | External Secrets Operator | Decoupled from pod lifecycle, supports multiple backends |
| Encryption at Rest | KMS envelope encryption on etcd | Protects against etcd compromise |
| Access Control | Kubernetes RBAC + Vault policies | Defense in depth |
| Secret Delivery | Filesystem mounts (tmpfs) | Avoids environment variable exposure |
| Auditing | Kubernetes audit policy + Vault audit backend | Full access trail |
This isn’t the only valid architecture. However, it resolves all three issues related to storage, access, and lifecycle without having to build a custom system to solve these problems.
Honest Recommendation – Who Needs What
If you are a small team on a managed cluster (EKS, GKE, AKS): You can begin with your cloud provider’s native secrets integration and ESO. This is because it incurs minimal operating costs and works for most workloads.
For sensitive data, compliance, or multi-cloud configurations, Vault is worth the operational investment. It’s more than sufficient because of audit logging and dynamic credential generation.
If you’re continuing to use Base64-encoded secrets in plain YAML in a repository: Stop. It’s the top priority to fix, regardless of anything else (what you do or don’t do).
Secrets management is no bed of roses in Kubernetes. It is not visible in the demo. Yet getting it wrong can have repercussions at inconvenient times.
I’m a technology writer passionate about AI and digital marketing. I create engaging and useful content that bridges the gap between complex technology concepts and digital technologies. My writing makes the process easy and engaging. I encourage participation I continue to research innovation and technology. Let’s connect and talk technology!
