Journey of Zone Aware Traffic

Sat, 18 Apr 2026 20:02:25 +0200

Today, I want to share a bit of details in our steps to support zoneaware traffic in our Kubernetes infrastructure.

You can learn a bit of Kubernetes and seeinteresting effects it can have. First I have to share something aboutthe environment.

Starting from zero!

There are basically two patterns in Kubernetes infrastructure deployments:

1 Kubernetes cluster per zone
1 Kubernetes cluster per region with multiple zones.

In our case it’s the latter, a single Kubernetes cluster spansmultiple zones, 3 by default. The advantage is that you have moresimple availability guarantees, because all your applications run across 3 zonesby default. On the other hand, doing traffic engineering, so preferclose instances, is more complex. Before I tell about the problem spaceand our findings, let me share our basic KubernetesIngresssetup. It is a 2-layer load balancer infrastructure serving a largescale microservice environment.

In German there is a proverb that says one picture is better thanthousands of words, so let’s see figure 1. It’s a bit outdated,because we run AWS network load balancer (NLB) instead of ALB, butrest looks the same today.

figure 1

We terminate TLS in the cloud load balancer and the HTTP routing isdone in our HTTP proxy skipper.Cloud load balancers are managed by kube-ingress-aws-controllerand DNS by external-dns.

We run this for about a decade since Kubernetes 1.3 and it evolvedquite a bit. If you want to lurk into our configuration, check it outin kubernetes-on-aws.

In this environment zone aware traffic has 4 parts:

client to cloud load balancer
cloud load balancer to skipper-ingress (skipper-ingress is the application based on skipper that is the data-plane and control-plane of our http proxy layer)
skipper-ingress to backend pods
cluster internal clients to skipper-ingress through a service type ClusterIP

Kube-ingress-aws-controller manages ALBs and NLBs, so 1. and 2. aredone, that one. You can set -nlb-cross-zone=false to disable sendingtraffic cross zone from the load balancer TargetGroup toskipper-ingress. If you set--nlb-zone-affinity=availability_zone_affinity all clients runningin the same zone in any AWS account will resolve DNS to same zone. Soinstead of 3 IPs for NLB Listeners you will resolve only one. It doesnot matter if you have a client in the same or in another cluster.

Sincev0.24.22skipper supports zone aware traffic (3.) in its kubernetesdataclient. Dataclients are the way to fetch data, that skipper usesto create its routing table. If you run skipper-ingress with thekubernetes dataclient configured, every skipper-ingress pod will fetchall relevant Kubernetes objects to build its routing tree. In largerenvironments this is quite some load on the Kubernetes control planeand it’s easy to break Kubernetes control plane by scaling out. Theskipper way to control the load targeting the Kubernetes control planeis to run skipper’s routesrv.Routesrvis a skipper control plane component. It uses the Kubernetes dataclient to fetchrouting information and exposes an API endpoint to fetch eskiproutes. Eskipis the skipper native routing language. Eskip has a set of paradigms:

Routes match by predicates
Predicates can be combined by && (logical AND), if you need a logical OR, you have to create another route.
Features in the request or response path are implemented in filters.
Backends are more than a list of load balancer pool members.

Eskip examples:

// syntaxrouteID: Predicate1 && Predicate2 -> filter1 -> filter2 -> <backend>;r1: Path("/resource/:id") -> setRequestHeader("X-Resource-Id", "${id}") -> clusterClientRatelimit("resource", 10, "1m", "X-Resource-Id") -> "https://backend.example.org";r2: Host("products.example.org") && Path("/products/:productId")    -> consistentHashKey("${productId}")    -> consistentHashBalanceFactor(1.25)    -> <consistentHash, "http://127.0.0.1:9998", "http://127.0.0.1:9997">;

Coming back to zone aware traffic.

If your data-plane skipper-ingress fetches routes from acontrol plane component like routesrv, the question is:

How does routesrv know where the skipper-ingress pod is running, that queries it?

Astonishing: routesrv has no idea!

Routesrv just exposes route API endpoints, that are used by thedata-plane pods to fetch routes:

/routes fetch all routes
/routes/:zone fetch zone aware routes

Skipper-ingress data plane pods know its zone by Kubernetes downwardsAPI. This makes it possible to pass Kubernetes metadata to the processby environment variables:

1env:2- name: KUBE_NODE_ZONE3  valueFrom:4    fieldRef:5      fieldPath: metadata.labels['topology.kubernetes.io/zone']

Now you can use the environment variable in a flag to skipper like-routes-urls=http://skipper-ingress-routesrv.kube-system.svc.cluster.local/routes/$(KUBE_NODE_ZONE)" to fetch zone aware routes.

Great, now we understand how we can do zone aware traffic 1.-3., but what about 4.?

Ok, 4. seems to be easy. You just plug an annotationservice.kubernetes.io/topology-mode: auto and kube-proxy will makesure your ClusterIP service is having a safe amount of Kubernetes podsin its layer 4 load balancer. We run this since more than 3 yearswithout an issue. Before this annotation there wasservice.kubernetes.io/topology-aware-hints annotation which didbasically the same thing. If you read thedocumentation,you see that there are some thoughts about safety, because you do notwant to create harm on a data-plane feature that every application hasto rely on.

As often in a life, things change. Sometimes implementation changes oryour monitoring adds more visibility. Your application requirements onthe infrastructure change or traffic patterns, because someonedeployed a new client to some service. Sometimes it’s a mixture of allof this visible in figure 2.

To understand better the zonal traffic in our clusters we created agraph how much requests by zone we have in skipper-ingressdata-plane. If we have some new data and it shows somethingunexpected, my first question is always: can we trust the data?Sometimes we do mistakes, like having a wrong unit or some selector isslightly wrong and you observe not exactly what you expected. So a bitof questioning your new data is always a great thing to find bugs.

However, in this case it seems we really were able to trust the data.During the last weeks we had some interesting effects in one of ourhigh traffic clusters. The effects are shown in figure 2. We can seethat within 1h there were 3 times a large share of throughput that hitonly one zone. After some minutes it was going back to normal.

figure 2 - RPS traffic by zone

What we see is flapping of the traffic distribution, which sometimescaused an unexpected latency spike to one of our applications. Thisflapping was caused by kube-proxy, that thought it might makes senseto flap between zone aware and not zone aware traffic. If you check safeguards,you can read:

4. One or more endpoints does not have a zone hint: When this  happens, the kube-proxy assumes that a transition from or to  Topology Aware Hints is underway. Filtering endpoints for a Service  in this state would be dangerous so the kube-proxy falls back to  using all endpoints.5. A zone is not represented in hints: If the kube-proxy is unable  to find at least one endpoint with a hint targeting the zone it  is running in, it falls back to using endpoints from all  zones. This is most likely to happen as you add a new zone into  your existing cluster.

Basically what we see in figure 2 is that if zone hints are populated,kube-proxy will write layer 4 rules such that rules are zone aware andwe have this large unbalanced traffic split. Later zone hintsdisappear or are not available for one or more endpoints (4.) and itdecides that it is too dangerous and the rules will change to non-zoneaware traffic. I don’t know why these hints disappear, but I did notfind any when I checked. One of the interesting facts are that we havethis flapping all the day for some weeks and it’s most often not anissue, but sometimes a latency spike up to 250ms happened and this islarge enough for high traffic low latency applications to fail.

After discussing this in Kubernetes sig-network community channel wewill try to switch to a more persistent service type ClusterIPconfigurations by using trafficDistribution: PreferSameZone, that isnow available in Kubernetes. It will provide no flapping for thetraffic distribution. I am looking forward to see the effects.

What about pods?

One other important configuration is that you have a balanced spreadof pods for clients, proxy and backends. This you can influence by settingtopologySpreadConstraints.The proxy and also application developers already enabled topologySpreadConstraints,example:

      topologySpreadConstraints:        - maxSkew: 1          topologyKey: topology.kubernetes.io/zone          whenUnsatisfiable: DoNotSchedule          labelSelector:            matchLabels:              component: ingress              application: skipper-ingress

This is configuration is important to not create too much of anunbalanced traffic by your clients. Also the targets of the proxy havealso to spread evenly so the horizontal pod autoscaling can do its joband keep the load of single pods in bounds.

One last thing

Of course Kubernetes would not be Kubernetes, that everyone loves andhates, if there would not be a missing feature. Beware about the factthat there is no zone aware down scaling.

Kubernetes infrastructure is super interesting, often details matterand sometimes there are missing features, that make you wonder, butall in all I am very happy with it.

If you have any questions or anything to share let me know in Mastodon.

Hugo Tests

Sat, 22 Nov 2025 19:55:23 +0100

Hello 👋

This is my first test post to test language rendering for Go and Bash and how to use pictures.

Something written in Go with highlights

1package main23import "fmt"45func main() {6    for i := 0; i < 3; i++ {7        fmt.Println("Value of i:", i)8    }9}

Another thing written in Go without highlights

1package foo23func Bar() (string, error) {4     println("Hello foo.Bar!")5     return "foo", nil6}

Now a little bit of bash:

 1function foo () { 2     echo "Hello Bash!" 3} 4 5cat <<EOF 6Hello World! 7EOF 8 9while [ 1 ]10do11        echo -n "."12        sleep 0.513done

A bit of goat

QR code:

A bit of mathjax

\[\begin{aligned}KL(\hat{y} || y) &= \sum_{c=1}^{M}\hat{y}_c \log{\frac{\hat{y}_c}{y_c}} \\JS(\hat{y} || y) &= \frac{1}{2}(KL(y||\frac{y+\hat{y}}{2}) + KL(\hat{y}||\frac{y+\hat{y}}{2}))\end{aligned}\]

A bit of inline mathjax $a^*=x-b^*$ and $\alpha \gt \beta$.

SVG with figure:

CPU flamegraph

PNG figure:

skipper logo

Tests

Resources

resources.Get (global assets){{ with resources.Get “perf.svg” }}{{ end }}

Resources.Get picture (per page) and try to render:

{{ $image := .Resources.Get “skipper.png” }}{{ with $image }}![skipper logo markdown styl does not work]({{ .RelPermalink }}){{ end }}

end:) O:)

Infrastructure on random thoughts