Incident on random thoughts

RFC9209 and the API of a 404

Thu, 09 Jul 2026 22:30:38 +0200

We have an interesting public debateon the status code 404 in Zalando’s API guidelines. I am a clearopponent to a change the proxy response from 404 to 5xx, but let’stry first to understand why they want to change.

The reason why this change proposal started was an incident, that ourcanaryhad some miss configuration. The canary is one pod in our Ingressfleet. This should mitigate the blast radius of updating our Ingressdata- and control-plane. However the data-plane started successful,but had no routes created by Kubernetes Ingressand RouteGroupobjects, so it responded 404 to all requests directed to applications.So as expected we had something like <5% error rate responding 404instead of reaching the applications, until some other control loopdetected that we have no active applications anymore. The mentionedcontrol loop got the connection to the skipper-ingress that respondedwith 404, which lead to downtime for all applications within oneminute. Basically the API for the control loop was, that status code404 meant “application does not exist” or even better “application wasdeleted”, so it needed to act on this change and clean up resources.

This is fine

Coming back to the pull request.

What do they want to change exactly?

Here is a cite, that shows what they want to change:

For instance, a Kubernetes Ingress router likehttps://github.com/zalando/skipper must be configured to returnthe status code {503} instead of {404}, if there is no route for anapplication service endpoint available.

What is 503 and how it is supposed to be used?

503 is defined inrfc9110and is the status code for StatusServiceUnavailable, which sounds legit atfirst. However citing the RFC clarifies the intention of status code 503:

The 503 (Service Unavailable) status code indicates that the server is currently unable to handle the request due to a temporary overload or scheduled maintenance, which will likely be alleviated after some delay. The server MAY send a Retry-After header field (Section 10.2.3) to suggest an appropriate amount of time for the client to wait before retrying the request.

If you check when 503 status code is used today in http proxies, youwill find for example circuit breaker, load shedder or in some casesif the proxy could not reach the backend. I think this is pretty muchinline with the RFC, for example load shedding and “temporaryoverload”. However a “route not found” case I don’t seerepresented. “Route not found” is not temporary, it’s theconfiguration, that might change after a deployment, but it is not theregular case.

Let’s understand the reasoning of the change proposal:

In particular, a middleware must not return a status code {404}, if theservice would have returned a different status code, as this could misleadclients into thinking that the resource does not exist when it actually does.

Note: In the case of the incident the proxy did know no backend service.

Clients can safely assume that {4xx} status codes are always authoritative andcomply with the end-to-end principle (see «256»). While a middleware canauthenticate a request and return {401} or {403} on behalf of the service, itmust never return a misleading status code, e.g. {404} or {410}, if the serviceis not configured or not responding.

They say that a client calling some URL should be able to use 404status code as the only thing it has to check for “an item does notexist”. So for https://example.org/items/foo reponse 404 means fooitem does not exist. This sounds also legit to me, even if I have adifferent opinion, because you can construct also weird examples likehttps://example.org/items/foo/bar/qux , that does not exist, even if/items/foo would exist. Normally you would get requests to some oldwordpress security vulnerability, so something like<some-path>/foo.php. And yes, in this case they want to have a 5xxstatus code.

What is the definition of 404 status code?

15.5.5. 404 Not FoundThe 404 (Not Found) status code indicates that the origin server did not find a current representation for the target resource or is not willing to disclose that one exists. A 404 status code does not indicate whether this lack of representation is temporary or permanent; the 410 (Gone) status code is preferred over 404 if the origin server knows, presumably through some configurable means, that the condition is likely to be permanent.A 404 response is heuristically cacheable; i.e., unless otherwise indicated by the method definition or explicit cache controls (see Section 4.2.2 of [CACHING]).

So a middlebox which is a proxy cache can heuristically cache the404 response. There is a status code 410 which should bepreferred for deleted resources. This would be in the case of theincident a possible way to differentiate between “deleted” and “doesnot exist” or “no matching route found”.

Why do I think a proxy that has no matching HTTP route for a givenHTTP request should return 404?

As a proxy to enable fast error detection you should indicate the mostlikely happening to guide your automations and investigators. Giventhat availability tools respond with 503. Given if a backend does notlisten to a port and Linux kernel return connection refused (oom of anapplication instance for example), which proxies will turn toa 502. Given that a severe error of the proxy or the application willbe a 500. If you serve public endpoints then you know, if you everlooked into your access logs that a quite big fraction of traffic aresecurity scanners, scanning all day for old vulnerabilities. As aproxy you repond by 404 and it’s all good.

Why should we change a 404 to a 5xx?

The proposal says:

a middleware must always return a temporary, non-authoritative server-side status code, such as:

{502} — Bad Gateway,
{503} — Service Unavailable, or
{504} — Gateway Timeout.

While the RFC for 404 says:

A 404 status code does not indicate whether this lack of representation is temporary or permanent

So 404 could be temporary, which does not contradict what themiddleware responds. Even worse a temporary 404 can not safelyassumed by the client that it is always authoritative. This means wehave another clear contradiction to the proposed arguments about theend-to-end semantics.

The proposal also mentions:

The end-to-end semantics of HTTP status codes is a fundamental part of the API contract.

Given that 404 could be temporary and the RFC shows 410 as a moreclear permanent status code, maybe the guideline should rather clarifythe use of 404. Even better it could tell about 410 instead.

Why do I think 404 is the right status code for HTTP proxies that donot find a matching route?

A route can be created, update and deleted by someone any time, sotemporary miss configured route returns 404 and if fixed again itreturns again the result as expected. This seems to be reasonablebased on the RFC.

Another more practical reason is that the amount of crawlers,vulnerability scanners, <bot-you-name-it> scan all the day your webpage every minute my tiny box hosting this blog gets traffic from someof these and I don’t care at all. Sometimes I name this “randominternet noise”. I also stopped logging them, because I don’t want towaste disk for this garbage. Even better our default in our Kubernetesingress layer for logs is to disable logs for 2xx, 3xx, 404 and429, because it’s a waste of disk, money and compute. Even betterless people are distracted because someone started a vulnerabilityscanner against a proper zero trust architecture with defaultauthentication all over the place. In the last 10 years running likethis there was no finding at all.

Here 1d of status codes for my blog.szuecs.net (enabled for one day):

$ awk '{print $9}' blog.szuecs.net/blog.szuecs.net_access.log.1 | sort | uniq -c    401 200     18 301      3 304    165 400   2226 404  <- 5x more than valid requests     17 405      1 408

So 5 times more traffic from vulnerability scanners, than fromgoogle-bot, bing-bot, … and you reading this post!

Another reason why I think it does not even matter to identify anissue like this via status code is, that with the raise of Opentracingand the proper standard Open Telemetry(short: OTel), provides a proper tool that easily identifes that thecall chain of a 404 response ended at the HTTP proxy in case you needto know who responded during a failure analysis.

Even developers that run their workload behind a proxy layer will wantto have a 404 instead of 5xx response, if you tell them that theiralerting of 5xx will create false alerts by random internet noise.

/EOF

Journey of Zone Aware Traffic

Sat, 18 Apr 2026 20:02:25 +0200

Today, I want to share a bit of details in our steps to support zoneaware traffic in our Kubernetes infrastructure.

You can learn a bit of Kubernetes and seeinteresting effects it can have. First I have to share something aboutthe environment.

Starting from zero!

There are basically two patterns in Kubernetes infrastructure deployments:

1 Kubernetes cluster per zone
1 Kubernetes cluster per region with multiple zones.

In our case it’s the latter, a single Kubernetes cluster spansmultiple zones, 3 by default. The advantage is that you have moresimple availability guarantees, because all your applications run across 3 zonesby default. On the other hand, doing traffic engineering, so preferclose instances, is more complex. Before I tell about the problem spaceand our findings, let me share our basic KubernetesIngresssetup. It is a 2-layer load balancer infrastructure serving a largescale microservice environment.

In German there is a proverb that says one picture is better thanthousands of words, so let’s see figure 1. It’s a bit outdated,because we run AWS network load balancer (NLB) instead of ALB, butrest looks the same today.

figure 1

We terminate TLS in the cloud load balancer and the HTTP routing isdone in our HTTP proxy skipper.Cloud load balancers are managed by kube-ingress-aws-controllerand DNS by external-dns.

We run this for about a decade since Kubernetes 1.3 and it evolvedquite a bit. If you want to lurk into our configuration, check it outin kubernetes-on-aws.

In this environment zone aware traffic has 4 parts:

client to cloud load balancer
cloud load balancer to skipper-ingress (skipper-ingress is the application based on skipper that is the data-plane and control-plane of our http proxy layer)
skipper-ingress to backend pods
cluster internal clients to skipper-ingress through a service type ClusterIP

Kube-ingress-aws-controller manages ALBs and NLBs, so 1. and 2. aredone, that one. You can set -nlb-cross-zone=false to disable sendingtraffic cross zone from the load balancer TargetGroup toskipper-ingress. If you set--nlb-zone-affinity=availability_zone_affinity all clients runningin the same zone in any AWS account will resolve DNS to same zone. Soinstead of 3 IPs for NLB Listeners you will resolve only one. It doesnot matter if you have a client in the same or in another cluster.

Sincev0.24.22skipper supports zone aware traffic (3.) in its kubernetesdataclient. Dataclients are the way to fetch data, that skipper usesto create its routing table. If you run skipper-ingress with thekubernetes dataclient configured, every skipper-ingress pod will fetchall relevant Kubernetes objects to build its routing tree. In largerenvironments this is quite some load on the Kubernetes control planeand it’s easy to break Kubernetes control plane by scaling out. Theskipper way to control the load targeting the Kubernetes control planeis to run skipper’s routesrv.Routesrvis a skipper control plane component. It uses the Kubernetes dataclient to fetchrouting information and exposes an API endpoint to fetch eskiproutes. Eskipis the skipper native routing language. Eskip has a set of paradigms:

Routes match by predicates
Predicates can be combined by && (logical AND), if you need a logical OR, you have to create another route.
Features in the request or response path are implemented in filters.
Backends are more than a list of load balancer pool members.

Eskip examples:

// syntaxrouteID: Predicate1 && Predicate2 -> filter1 -> filter2 -> <backend>;r1: Path("/resource/:id") -> setRequestHeader("X-Resource-Id", "${id}") -> clusterClientRatelimit("resource", 10, "1m", "X-Resource-Id") -> "https://backend.example.org";r2: Host("products.example.org") && Path("/products/:productId")    -> consistentHashKey("${productId}")    -> consistentHashBalanceFactor(1.25)    -> <consistentHash, "http://127.0.0.1:9998", "http://127.0.0.1:9997">;

Coming back to zone aware traffic.

If your data-plane skipper-ingress fetches routes from acontrol plane component like routesrv, the question is:

How does routesrv know where the skipper-ingress pod is running, that queries it?

Astonishing: routesrv has no idea!

Routesrv just exposes route API endpoints, that are used by thedata-plane pods to fetch routes:

/routes fetch all routes
/routes/:zone fetch zone aware routes

Skipper-ingress data plane pods know its zone by Kubernetes downwardsAPI. This makes it possible to pass Kubernetes metadata to the processby environment variables:

1env:2- name: KUBE_NODE_ZONE3  valueFrom:4    fieldRef:5      fieldPath: metadata.labels['topology.kubernetes.io/zone']

Now you can use the environment variable in a flag to skipper like-routes-urls=http://skipper-ingress-routesrv.kube-system.svc.cluster.local/routes/$(KUBE_NODE_ZONE)" to fetch zone aware routes.

Great, now we understand how we can do zone aware traffic 1.-3., but what about 4.?

Ok, 4. seems to be easy. You just plug an annotationservice.kubernetes.io/topology-mode: auto and kube-proxy will makesure your ClusterIP service is having a safe amount of Kubernetes podsin its layer 4 load balancer. We run this since more than 3 yearswithout an issue. Before this annotation there wasservice.kubernetes.io/topology-aware-hints annotation which didbasically the same thing. If you read thedocumentation,you see that there are some thoughts about safety, because you do notwant to create harm on a data-plane feature that every application hasto rely on.

As often in a life, things change. Sometimes implementation changes oryour monitoring adds more visibility. Your application requirements onthe infrastructure change or traffic patterns, because someonedeployed a new client to some service. Sometimes it’s a mixture of allof this visible in figure 2.

To understand better the zonal traffic in our clusters we created agraph how much requests by zone we have in skipper-ingressdata-plane. If we have some new data and it shows somethingunexpected, my first question is always: can we trust the data?Sometimes we do mistakes, like having a wrong unit or some selector isslightly wrong and you observe not exactly what you expected. So a bitof questioning your new data is always a great thing to find bugs.

However, in this case it seems we really were able to trust the data.During the last weeks we had some interesting effects in one of ourhigh traffic clusters. The effects are shown in figure 2. We can seethat within 1h there were 3 times a large share of throughput that hitonly one zone. After some minutes it was going back to normal.

figure 2 - RPS traffic by zone

What we see is flapping of the traffic distribution, which sometimescaused an unexpected latency spike to one of our applications. Thisflapping was caused by kube-proxy, that thought it might makes senseto flap between zone aware and not zone aware traffic. If you check safeguards,you can read:

4. One or more endpoints does not have a zone hint: When this  happens, the kube-proxy assumes that a transition from or to  Topology Aware Hints is underway. Filtering endpoints for a Service  in this state would be dangerous so the kube-proxy falls back to  using all endpoints.5. A zone is not represented in hints: If the kube-proxy is unable  to find at least one endpoint with a hint targeting the zone it  is running in, it falls back to using endpoints from all  zones. This is most likely to happen as you add a new zone into  your existing cluster.

Basically what we see in figure 2 is that if zone hints are populated,kube-proxy will write layer 4 rules such that rules are zone aware andwe have this large unbalanced traffic split. Later zone hintsdisappear or are not available for one or more endpoints (4.) and itdecides that it is too dangerous and the rules will change to non-zoneaware traffic. I don’t know why these hints disappear, but I did notfind any when I checked. One of the interesting facts are that we havethis flapping all the day for some weeks and it’s most often not anissue, but sometimes a latency spike up to 250ms happened and this islarge enough for high traffic low latency applications to fail.

After discussing this in Kubernetes sig-network community channel wewill try to switch to a more persistent service type ClusterIPconfigurations by using trafficDistribution: PreferSameZone, that isnow available in Kubernetes. It will provide no flapping for thetraffic distribution. I am looking forward to see the effects.

What about pods?

One other important configuration is that you have a balanced spreadof pods for clients, proxy and backends. This you can influence by settingtopologySpreadConstraints.The proxy and also application developers already enabled topologySpreadConstraints,example:

      topologySpreadConstraints:        - maxSkew: 1          topologyKey: topology.kubernetes.io/zone          whenUnsatisfiable: DoNotSchedule          labelSelector:            matchLabels:              component: ingress              application: skipper-ingress

This is configuration is important to not create too much of anunbalanced traffic by your clients. Also the targets of the proxy havealso to spread evenly so the horizontal pod autoscaling can do its joband keep the load of single pods in bounds.

One last thing

Of course Kubernetes would not be Kubernetes, that everyone loves andhates, if there would not be a missing feature. Beware about the factthat there is no zone aware down scaling.

Kubernetes infrastructure is super interesting, often details matterand sometimes there are missing features, that make you wonder, butall in all I am very happy with it.

If you have any questions or anything to share let me know in Mastodon.

Hugo Tests

Sat, 22 Nov 2025 19:55:23 +0100

Hello 👋

This is my first test post to test language rendering for Go and Bash and how to use pictures.

Something written in Go with highlights

1package main23import "fmt"45func main() {6    for i := 0; i < 3; i++ {7        fmt.Println("Value of i:", i)8    }9}

Another thing written in Go without highlights

1package foo23func Bar() (string, error) {4     println("Hello foo.Bar!")5     return "foo", nil6}

Now a little bit of bash:

 1function foo () { 2     echo "Hello Bash!" 3} 4 5cat <<EOF 6Hello World! 7EOF 8 9while [ 1 ]10do11        echo -n "."12        sleep 0.513done

A bit of goat

QR code:

A bit of mathjax

\[\begin{aligned}KL(\hat{y} || y) &= \sum_{c=1}^{M}\hat{y}_c \log{\frac{\hat{y}_c}{y_c}} \\JS(\hat{y} || y) &= \frac{1}{2}(KL(y||\frac{y+\hat{y}}{2}) + KL(\hat{y}||\frac{y+\hat{y}}{2}))\end{aligned}\]

A bit of inline mathjax $a^*=x-b^*$ and $\alpha \gt \beta$.

SVG with figure:

CPU flamegraph

PNG figure:

skipper logo

Tests

The product of {{ $v1 }} and {{ $v2 }} is {{ mul $v1 $v2 }}.

Resources

resources.Get (global assets){{ with resources.Get “perf.svg” }}{{ end }}

Resources.Get picture (per page) and try to render:

{{ $image := .Resources.Get “skipper.png” }}{{ with $image }}![skipper logo markdown styl does not work]({{ .RelPermalink }}){{ end }}

end:) O:)