RBAC and namespaces - deep dive

RBAC is the boundary between "we have a cluster" and "we have a multi-tenant production cluster." Most clusters I have seen in the wild are either wide open (cluster-admin handed out like candy) or so locked down nobody can ship. The middle path requires thinking about identity, scope, and the actual operations each workload needs.

Namespaces: what they are and what they are not

Namespaces give you:

• A naming scope for namespaced resources.
• A target for RBAC RoleBindings.
• A target for NetworkPolicies (which can select pods by namespace).
• A target for ResourceQuotas and LimitRanges.
• A target for default-deny PodSecurity policies.

Namespaces do NOT give you:

• Network isolation. By default any pod can talk to any other pod cluster-wide. You need NetworkPolicy (and a CNI that enforces it) for that.
• Resource isolation. Nodes are shared. A noisy pod in namespace A can starve namespace B unless you use ResourceQuota and node-level isolation (taints, dedicated node groups).
• True security isolation. Cluster-scoped resources, shared kernel, shared etcd. For hard tenancy, use separate clusters or virtual clusters (vcluster).

Namespace patterns I have seen work

• Per environment in non-prod: dev, staging, qa in a shared non-prod cluster.
• Per team in shared infra: team-payments, team-search, with quotas and network policies.
• Per app stack: monitoring, ingress, cert-manager for cluster add-ons.
• Per tenant for SaaS: only viable for trusted tenants because of soft isolation; for true multi-tenancy use separate clusters.

Avoid: one giant default namespace, one namespace per microservice (you will hate yourself at 200 namespaces), or no namespaces at all.

RBAC: the four resources, in depth

Role vs ClusterRole

A Role grants verbs on resources in its own namespace. A ClusterRole can:

• Grant verbs on cluster-scoped resources (Nodes, PVs, StorageClasses).
• Grant verbs on namespaced resources across all namespaces (when bound via ClusterRoleBinding).
• Be referenced by a RoleBinding to grant its verbs within one namespace (reuse pattern).

The reuse pattern is huge. Define a ClusterRole called app-developer once. Bind it via RoleBinding in each team namespace. Update permissions in one place.

Aggregated ClusterRoles

You can compose ClusterRoles using label selectors. The built-in view, edit, admin ClusterRoles are aggregations. Your custom CRDs can contribute permissions to these by labeling a new ClusterRole with rbac.authorization.k8s.io/aggregate-to-view: "true". Powerful for plugin patterns.

Verbs you forget exist

• bind and escalate: required to create RoleBindings that grant permissions you do not already have. Without these, even a cluster-admin-like role cannot grant cluster-admin (privilege escalation prevention).
• impersonate: lets a subject act as another user, group, or SA. Used by kubectl --as. Tightly controlled.
• Subresources: pods/exec, pods/log, pods/portforward, pods/attach. Granting pods/get does not grant pods/exec. People conflate these and break access.

ServiceAccounts and Pod identity

Every Pod authenticates to the API server as a ServiceAccount. If unspecified, it is the default SA in its namespace. The token is mounted at /var/run/secrets/kubernetes.io/serviceaccount/token along with the cluster CA and namespace.

Since 1.22, tokens are projected, time-bound, and audience-scoped (BoundServiceAccountTokenVolume). Tokens expire after an hour by default and rotate automatically. Long-lived Secret-based SA tokens are deprecated.

Default SAs should have no permissions. They cannot have permissions removed (they have none by default) but they exist in every namespace and are the fallback. If your app does not need to talk to the API server, set automountServiceAccountToken: false on the Pod or the SA. Saves a token mount and removes one attack vector.

IRSA: bridging Kubernetes SAs to AWS IAM

On EKS the production pattern is IAM Roles for Service Accounts (IRSA).

Flow:

1. EKS exposes an OIDC issuer URL.
2. You create an IAM OIDC provider trusting that issuer.
3. You create an IAM role with a trust policy that allows AssumeRoleWithWebIdentity from that OIDC provider, scoped to a specific SA in a specific namespace.
4. You annotate the SA: eks.amazonaws.com/role-arn: arn:aws:iam::123:role/MyAppRole.
5. The Pod gets a projected token; AWS SDK auto-detects and calls STS to swap it for AWS credentials.

The wins: no long-lived AWS credentials in the cluster, granular IAM per SA, audit trail in CloudTrail. At Binocs this was how every app got its AWS access - S3 buckets, KMS keys, DynamoDB tables, RDS IAM auth.

Pod Identity (newer, 2023) is the AWS-native replacement that does not require OIDC provider setup. Easier to manage at scale. Either works.

RBAC design patterns

Least privilege ServiceAccounts

For each Deployment, create a dedicated SA. Bind it to a Role that grants exactly the permissions the app needs. Default to nothing. Add permissions when the app actually fails because of missing access.

Read-only humans, write via CI

Developers get view ClusterRole bound in their team namespaces. Maybe edit in dev/staging. Production writes happen through CI/CD pipelines that use a dedicated SA with scoped permissions. No human directly applies to prod.

Break-glass admin

A dedicated cluster-admin ClusterRoleBinding for a specific Group that is normally empty. SSO grants the Group membership only on PagerDuty trigger or explicit approval. Audit log every access.

Namespace owner pattern

Each team namespace has an admin RoleBinding for the team's Group, granting admin ClusterRole inside that namespace. Teams self-manage RBAC for their own apps within their namespace. Cluster admins do not have to be in the loop for every change.

Auditing RBAC

The single best tool: kubectl auth can-i.

kubectl auth can-i delete pods --as=system:serviceaccount:default:my-app -n payments

For broader audit:

• rakkess lists all permissions for a subject.
• kubectl-who-can (krew plugin) reverses it: who can do X.
• audit2rbac analyzes audit logs to generate minimal RBAC.

For ongoing security, Falco, kube-bench, and kube-hunter scan for misconfigurations.

PodSecurity admission (PSP replacement)

PodSecurityPolicies were removed in 1.25. The replacement is Pod Security admission, a built-in controller that enforces one of three levels per namespace via labels:

• pod-security.kubernetes.io/enforce: restricted - hardened, no root, no privileged.
• pod-security.kubernetes.io/enforce: baseline - sensible defaults.
• pod-security.kubernetes.io/enforce: privileged - no restrictions.

Label every namespace. For richer policy (anything beyond the three built-in levels), use Kyverno or OPA Gatekeeper.

What I ran at Binocs

• One namespace per environment in non-prod, one cluster per environment in prod.
• One SA per Deployment, scoped Role, never the default SA.
• IRSA for every app's AWS access. No long-lived AWS creds anywhere.
• Pod Security admission set to restricted enforce in all app namespaces. baseline for system add-ons that needed it.
• ResourceQuotas per team namespace to prevent one runaway team from eating the cluster.
• Audit policy logging Secret reads, exec into pods, RBAC changes. Shipped to CloudWatch with alerts on anomalous patterns.

The IRSA migration replaced about 15 long-lived IAM users we had been mounting as Kubernetes Secrets. That cleanup was as valuable as any of the cost work. Compromise blast radius dropped from "all our AWS" to "one Pod's narrow permissions."

The interview narrative

Frame namespaces as the scoping primitive (not isolation). Walk through RBAC's four resources: Role and ClusterRole are permission definitions, RoleBinding and ClusterRoleBinding grant them to subjects. ServiceAccounts are Pod identity. On EKS, IRSA bridges Kubernetes SA identity to AWS IAM with no long-lived creds. Hit least privilege, the default-SA trap, and Pod Security admission as the three production patterns. If they go deep, talk about audit tooling and the break-glass pattern for human admin access.