Page 3 of 4
Building an Internal Developer Platform (IDP) means treating "provision me a working environment/service" as a product, not a ticket queue - golden paths (an opinionated, supported way to deploy a standard service) trade some flexibility for dramatically reduced cognitive load on the majority of teams who don't need anything exotic.
Self-service infrastructure and namespace-as-a-product patterns typically layer CRDs + Operators + policy engines (see below) to let application teams provision what they need (a namespace, a database, a queue) through a constrained, safe abstraction - a CRD like kind: Environment that a controller turns into a namespace, RBAC, resource quotas, and network policy, all consistently, rather than every team hand-rolling their own.
Multi-tenancy at the platform level is fundamentally a tradeoff between isolation and operational overhead - the platform engineering job is choosing where on that spectrum each tenant class actually needs to sit, not maximizing isolation everywhere by default.
Soft multi-tenancy (trusted tenants, isolated mainly by convention and RBAC/namespaces) versus hard multi-tenancy (untrusted or mutually adversarial tenants, requiring much stronger isolation - separate node pools, gVisor/Kata Containers-style sandboxing, or genuinely separate clusters) is the framing question every multi-tenant Kubernetes design has to answer honestly before anything else.
Real tenant isolation is layered, not one setting: namespace isolation (RBAC scoped per namespace) + network isolation (default-deny NetworkPolicy per tenant namespace) + resource isolation (ResourceQuota/LimitRange preventing one tenant from starving others) + Pod Security (preventing privilege escalation that could cross tenant boundaries). Skipping any one layer under the assumption "the others cover it" is exactly how multi-tenant isolation quietly fails in practice.
OPA (Open Policy Agent) and its Kubernetes-native front-ends Gatekeeper and Kyverno enforce policy as validating/mutating admission webhooks - "every image must come from our internal registry," "every Deployment must set resource limits," "no Pod may run as root" - rejected or auto-corrected at admission time, before the object is ever written to etcd, rather than caught after the fact by a scanning job.
Policy as code means these rules live in version control, get reviewed like any other change, and are tested before rollout - the same discipline applied to infrastructure change management generally, just scoped to admission-time enforcement specifically. A policy engine misconfigured too aggressively is a real production incident vector in its own right: a bad default-deny policy rule can block all deployments cluster-wide the moment it's applied, which is why staged rollout (dry-run/audit mode before enforce mode) is standard practice for any new policy.
Supply-chain security treats "what's actually running in this cluster" as a chain of trust starting well before deployment: image signing (Cosign) cryptographically ties an image to its build provenance, an SBOM (Software Bill of Materials) enumerates exactly what's inside an image, and admission enforcement can reject any image that isn't signed or doesn't meet policy - closing the gap between "we scanned this image once" and "we can prove what's actually running right now is what we think it is."
Runtime security (Falco being the standard open-source tool) monitors actual running behavior for anomalies - a container unexpectedly spawning a shell, writing to a sensitive path, making an unexpected outbound connection - catching exploitation after something has already gotten past every earlier control, which is why it's a distinct, necessary layer rather than redundant with admission-time policy.
Zero-trust Kubernetes (workload identity + mTLS between services, typically via a service mesh) removes the assumption that "inside the cluster network" implies "trusted" - every service authenticates every connection cryptographically, so a compromised Pod can't simply talk to anything reachable on the network the way flat, unauthenticated Pod-to-Pod networking otherwise allows by default.
At real scale, each control-plane component has its own specific pressure point: the API server's request rate is governed by QPS/Burst settings (both server-side and per-client), and a client hammering the API server without respecting these will get throttled rather than degrading the server for everyone else - which is exactly the point of the limit. etcd performance degrades with total object count and write rate long before disk space runs out, which is why compaction/defragmentation (see etcd - Expert) is a genuine performance lever, not just housekeeping.
DNS performance at scale is a frequently-underestimated bottleneck: every Pod's DNS resolution path (including failed lookups retried across a multi-entry search domain list) adds up fast across thousands of Pods, and CoreDNS itself needs real resource allocation and caching tuning proportional to cluster size, not the defaults appropriate for a small cluster.
Large-cluster resource tuning generally means: raising API server QPS/Burst limits deliberately (not just to make errors go away), scaling etcd hardware ahead of object count growth, and treating CNI/DNS capacity as first-class scaling dimensions alongside compute - the same categories of pressure as a small cluster, just where the defaults stop being adequate.
Real scenario-based DevOps questions, hands-on practice, and clear explanations for every answer.