ConfigMaps and Secrets - deep dive

Configuration and secrets are where almost every "works locally, breaks in prod" bug lives. The Kubernetes model is opinionated but not always obvious. Get the consumption model right (env vs file, snapshot vs live), get the security model right (KMS, secret manager, RBAC), and the rest is plumbing.

The shape

Both objects are tiny:

apiVersion: v1
kind: ConfigMap
metadata: { name: app-config }
data:
  LOG_LEVEL: "info"
  FEATURE_X: "true"
---
apiVersion: v1
kind: Secret
metadata: { name: db-creds }
type: Opaque
data:
  PASSWORD: cGFzc3dvcmQxMjM=  # base64 of "password123"

data is base64 in Secrets so binary values are safe. stringData lets you write plain text in Secrets and the API server encodes it. Both objects have a 1 MiB size limit (etcd object limit).

How Pods consume them

Environment variables

envFrom:
  - configMapRef: { name: app-config }
  - secretRef: { name: db-creds }

Snapshot at Pod start. Updates to the underlying ConfigMap/Secret do not propagate until the Pod restarts. This is fine for things like log level if you accept a rolling restart to change them.

Volume mounts

volumes:
  - name: config
    configMap: { name: app-config }
  - name: secrets
    secret: { secretName: db-creds }
volumeMounts:
  - { name: config, mountPath: /etc/config }
  - { name: secrets, mountPath: /etc/secrets, readOnly: true }

Each key becomes a file. The kubelet syncs the projected volume from the latest API server state on each kubelet sync cycle (default 60s, plus configMapAndSecretChangeDetectionStrategy). Your app needs to re-read the file or use a file-watcher to pick up changes.

subPath mount

volumeMounts:
  - { name: config, mountPath: /etc/app.conf, subPath: app.conf }

Mounts a single key as a file at that exact path. Useful when you need to put the config file at a path the app expects, not in a directory. Trade-off: subPath mounts do NOT receive updates. Pod restart required.

Updating without restarting

If you mount as a volume, updates propagate. The trick is making your app reload. Options:

1. File watcher in the app: inotify on /etc/config. Most stable.
2. Sidecar that reloads: configmap-reload sidecar sends SIGHUP to your process on file change. Works for NGINX, fluent-bit.
3. stakater/Reloader controller: watches ConfigMaps and Secrets, triggers a rolling restart of any Deployment annotated to depend on them. Pragmatic when your app cannot hot-reload.

For env-injected config, the only path is a Pod restart. Reloader is the simplest fix.

Secrets security: the full story

Default Kubernetes Secrets are just base64. Anyone with:

• get secrets RBAC permission in the namespace can read them.
• Read access to etcd can dump them (anyone with control plane access).
• Pod exec in a Pod that mounts them can read the env or files.

To make Secrets actually secret you layer four things:

1. Encryption at rest

Configure the API server with an EncryptionConfiguration that wraps Secrets with a KMS-backed key. On EKS, enable envelope encryption with a KMS key at cluster creation. Each Secret is encrypted with a DEK, the DEK is encrypted with the KMS CMK, only the encrypted blobs are in etcd. Compromise of etcd no longer compromises Secrets.

2. RBAC scoping

Most apps do not need a wildcard get secrets. Bind a ServiceAccount to a Role that lists only the specific Secret names it needs. The default default SA in a namespace should never have Secret read access.

3. Audit logging

Log all get/list/watch on Secrets in your audit policy. Stream to CloudWatch / S3. You want to know if anything weird is reading secrets.

4. External secret manager

Do not put production secrets in Git, even encrypted. Source of truth lives in AWS Secrets Manager / Vault / GCP Secret Manager. Kubernetes is downstream.

External Secrets Operator (ESO) pattern

ESO is the standard pattern for syncing from an external secret store.

You write an ExternalSecret CR that says "fetch this key from this SecretStore, write a Kubernetes Secret named X with these mapped keys, refresh every 1h." ESO does the rest. Rotation in AWS Secrets Manager propagates to the Kubernetes Secret on the next refresh.

Alternatives in the same space:

• CSI Secrets Store Driver: mount secrets directly from the store as files, no Kubernetes Secret object. Cleaner blast radius (compromise of the cluster does not expose secrets in etcd). More complex setup.
• Vault Agent Injector: Vault-specific. Mutating webhook injects a sidecar that pulls and templates secrets to a shared volume.

For most teams, ESO is the right starting point. CSI driver for compliance-heavy workloads.

GitOps and secrets: SOPS or sealed-secrets

If you must store secrets in Git (single source of truth for GitOps purists):

• Mozilla SOPS: encrypts values in YAML using a KMS key. The encrypted YAML is safe in Git. CI decrypts and applies.
• Bitnami sealed-secrets: a controller in the cluster holds a private key. You encrypt with the public key, commit the SealedSecret, the controller decrypts to a Secret in cluster. Cluster-bound.

Both work. ESO is still better in production because rotation lives outside Git.

ConfigMap practicalities

A few details that bite:

• Immutable ConfigMaps and Secrets (immutable: true) speed up the cluster significantly at scale. Without immutability the kubelet watches every mounted ConfigMap for changes. With immutability it does not. For known-static config use immutable.
• Versioned ConfigMaps: a common pattern is to name your ConfigMap app-config-v1, app-config-v2 and reference the new name in the Deployment. Roll forward by changing the reference, rollback by reverting. Old ConfigMaps stay around for instant rollback. Avoids the "mutate in place and surprise everyone" problem.
• ConfigMap as JSON/YAML blob: instead of N keys, store a single config.json key with the structured config. Easier to template, your app already has a JSON parser.

What I ran at Binocs

• ConfigMaps for environment-specific config (region, log level, feature flags). Mounted as env vars for the static stuff, as files for things that needed reload.
• AWS Secrets Manager as source of truth for DB credentials, third-party API keys, signing keys.
• External Secrets Operator with IRSA (IAM Roles for ServiceAccounts) so each app's ServiceAccount could only fetch its own prefix in Secrets Manager.
• KMS encryption enabled at the EKS control plane.
• Reloader controller annotated on the few Deployments that needed config-driven restarts.

The IRSA + ESO + Secrets Manager combo is the production-ready stack. Setup is a day. Operational cost is near zero.

The interview narrative

Lead with the shape: ConfigMap and Secret are key-value, consumed as env vars (snapshot) or volume mounts (live update). Secrets are base64, not encrypted. For production you need KMS encryption at rest, tight RBAC, audit logging, and a real secret manager (AWS Secrets Manager + ESO with IRSA) so rotation happens upstream. Throw in Reloader for the "config change forces restart" case and you have covered everything they will ask.