Page 2 of 2
Once a system spans regions, you need a way to route each user to the right region, and there are two complementary mechanisms for that. GeoDNS answers DNS queries differently depending on where the query is coming from, returning the IP address of the nearest region - simple to reason about and works with standard DNS infrastructure, but it inherits the TTL propagation delay discussed in the hard tier, and its accuracy depends on the resolver's location, not necessarily the actual end user's.
Anycast takes a different approach entirely: the exact same IP address is announced from multiple physical locations at once, and the network's own routing (BGP) delivers a packet to whichever announcing location is topologically closest - no DNS involved in the region decision at all, and failover is near-instant since it's handled at the routing layer rather than depending on any TTL expiring. The tradeoff is operational: anycast requires control over BGP announcements, which is a level of network infrastructure most teams access only through a cloud provider's edge network product rather than running themselves, and debugging "which location actually served this request" is inherently less transparent than reading a DNS answer.
The CAP theorem states that when a network partition happens - some part of the network can't communicate with another part - a distributed system has to choose between consistency (every node sees the same data) and availability (every request gets a response), because it genuinely cannot guarantee both during that partition. This isn't an abstract theorem to memorize, it's a decision that gets made, explicitly or by accident, in every multi-region or multi-node system you design.
The concrete question to ask is: when a network partition happens (and at meaningful scale, it eventually will), should the system stay available and risk serving stale or conflicting data, or should it refuse requests to guarantee correctness? Neither answer is universally right - a payments ledger has very different tolerance for stale data than a product recommendation cache does. What separates deliberate design from an accidental outcome is making this choice explicitly, up front, for each part of the system, rather than discovering your system's actual answer for the first time during a real partition, under pressure, when it's too late to choose.
At large organizations, individual teams generally shouldn't need to become networking experts just to ship a service - that knowledge gets consolidated into a platform team that builds paved-road tooling and abstractions on top of raw networking primitives. This might mean an internal service mesh configured with sane defaults, self-service TLS certificate provisioning wired into the certificate management already discussed, or standardized ingress patterns every team can plug into without hand-rolling load balancer and DNS configuration each time.
The judgment call underneath all of it is where to draw the abstraction boundary. Abstract too little, and every team reinvents the same networking decisions - and their mistakes - independently, which doesn't scale organizationally. Abstract too much, and teams lose the ability to make legitimate tradeoffs specific to their own service's actual needs, fighting the platform instead of being helped by it. Good internal platform tooling makes the common case genuinely easy by default, while still leaving an intentional, well-documented escape hatch for the real exceptions - rather than either forcing everyone through one rigid path or leaving every option wide open with no guidance at all.
Compute and storage capacity planning gets a lot of attention; network capacity is just as real a constraint and gets far less. Bandwidth between regions, connection limits on load balancers, DNS query volume, and NAT gateway throughput all have hard ceilings, and unlike adding another application server, scaling core network infrastructure often can't be done instantly - some of it involves real provisioning lead time with a cloud provider, not just an autoscaling policy kicking in.
The practice that matters is treating network capacity as a first-class input to capacity planning, not an assumed-infinite backdrop behind it: modeling peak traffic against actual link and gateway limits, load-testing against realistic multi-region traffic patterns rather than only single-region load, and specifically planning for what a failover event does to per-region network load - because failover, by definition, concentrates traffic that was previously spread across multiple regions onto fewer of them, which is exactly the moment you least want to discover a network-layer bottleneck you hadn't modeled for.
Real scenario-based DevOps questions, hands-on practice, and clear explanations for every answer.