Engineering blog
Becoming ready for the CVE storm
By Caesar
By the end of October, 89,859 CVEs had been published in 2026. All of 2025 had 48,151. October alone brought 17,060, a third of last year in one month. Much of the growth is credited to automated, AI-assisted vulnerability research, and it is not slowing down. Jerry Gamblin, who publishes monthly statistics from the NVD, put the other half of the picture in his mid-year check-in: of the CVEs from the first half of the year, 0.24% had made CISA’s list of vulnerabilities known to be exploited.
Nobody patches almost 300 CVEs a day, and almost none of them will matter to All-Chat. The ones that do matter need a fix I can actually install, quickly. That only works for software that still gets fixes. A project that has been archived, or a version past its end of life, gets no fix no matter how serious the CVE is, and then the fix is a migration instead of an update. So before the storm reaches something I run, I wanted to know how much of what I run is still supported. I took one day, 2026-11-01, to find out and to fix what I found.
Taking stock
The inventory had to cover everything that serves allch.at, not just the code in the All-Chat repository: the ingress, the databases, the Kubernetes cluster underneath, the monitoring, and every container image running in the namespaces the app depends on. I listed every running container by the exact image it runs, scanned each image with Syft into a software bill of materials, and checked every operating system, runtime and application version in it against endoflife.date. For projects that site does not track, I checked whether the upstream repository was archived and how far behind the latest release I was.
That came to 81 images and 12,897 components, with 49 end-of-life findings and 15 more that were well behind. Some of them were in places that matter most:
- The ingress controller that terminates every request to allch.at, ingress-nginx, was retired and its repository archived in March. The nginx inside it had been past its end of life for over a year.
- Kubernetes itself was on 1.33, which stopped getting patches at the end of June. The container runtime under it had followed in July.
- The PostgreSQL image under every database was eleven patch releases behind and built on Debian 11, whose support ended in August. The operator that runs those databases was on a release that went out of support in May 2025.
- The object storage behind media.allch.at ran on MinIO, whose repository was archived in April, from a frozen image that gets no rebuilds.
- The certificate manager was on a release from early 2024. Argo CD, which deploys everything, Prometheus and Grafana were all past their support windows.
- Alpine 3.21, still under one of my own services and a few third-party images, reached its end of life on the day I ran the audit.
What changed
The front door. ingress-nginx is replaced by Envoy Gateway and the Gateway API: one shared gateway, a wildcard certificate per domain, and a route per app. All 20 old routing rules had to be rewritten, including CORS, request size limits, long-lived WebSocket timeouts and the single sign-on in front of the admin tools. Before moving any traffic, I compared every host through the new gateway against the live old one, status code by status code, then moved the public addresses over in one change. No request failed with a server error in the minutes after.
Certificates. cert-manager went from 1.14 to 1.21, one minor version at a time. Certificates are now issued through a DNS challenge at Cloudflare instead of an HTTP request through the proxy, so the proxy can no longer get in the way of renewals.
Databases. Before touching anything, I took a full dump of every database. The database operator went from 1.24 to 1.30, one minor version at a time. It can swap its own agent inside a running database pod, so six upgrades of the operator restarted PostgreSQL zero times. On the way I found a setting that told the operator to watch only one namespace. It had never taken effect, because the old version read it too late, but a newer version reading it in time would have stopped managing five of the six database clusters. It is gone. All six clusters then moved to PostgreSQL 17.11 on Debian 13.
Kubernetes. The cluster went from 1.33 to 1.36 in three steps, because Kubernetes upgrades one minor version at a time. Each step upgraded the six nodes one by one, control plane first, with a snapshot of the cluster state before every run. That is eighteen node restarts, none of them failed, and nothing that runs on the cluster noticed.
Everything else. Every All-Chat service now builds on Go 1.27.3 and Alpine 3.24 (#1129). The object storage runs on Silo, a maintained fork of MinIO that reads the same data; I checked that on a copy written by the exact old build before switching. Argo CD, Prometheus, Grafana, Loki and the rest of the monitoring are on current releases, and the tracing backend went from seven components to one.
What went wrong on the way
Upgrading this much in one day on a live system caused two problems that were noticeable from the outside.
Redis briefly had two masters. Redis carries chat on its way from the platforms to your overlay. I updated its image, and the nodes and the watchers that decide which node is in charge restarted at the same time. The restarting watchers had forgotten who the master was, and one node came back as a second master that the others did not know about. For about eight minutes, most of the connections went to that node, where every write was refused and messages published on the real master never arrived, so some overlays missed chat. The watchers now finish restarting before any node does, each node waits before the next one restarts, and the proxy in front of Redis drops connections to a node that stops being master.
A deploy at the wrong hour. The last change of the day reduced how often the two TikTok listeners hand rooms back and forth (#1132). I deployed it on a Sunday evening, when many streamers are live. A deploy restarts both listeners, so every live TikTok room reconnected at once, which ran into the hourly connection budget the last post was about. Nine rooms were parked for up to an hour. TikTok listener changes now wait for a quiet hour.
The cost
- The two incidents above. Upgrading less in one sitting would have made each one less likely, and stretching the work over a week would have made the gap to supported versions last a week longer. I chose the day.
- Traces recorded before the switch to the new tracing backend are gone, and so is the metrics history from before today. Metrics had been kept on disk that did not survive a restart, which is fixed now.
- Five end-of-life findings remain. They are in third-party images whose maintainers have not shipped anything newer yet, or in old copies of an image that disappear the next time those services restart.
Where it stands
The same scan in the evening found 74 images, 5 end-of-life findings instead of 49, and 1 outdated component instead of 15. None of the remaining end-of-life ones are on the path a chat message takes.
What is still open:
- Kubernetes 1.37 is out, but Envoy Gateway does not support it yet. The upgrade is one command per step now, and it waits for that release.
- The scan checks support status, not CVEs. Running it regularly, and alerting on new end-of-life findings, is what keeps today from having to be repeated in a year.
The lesson I took: the number of CVEs is not something I can do anything about. What I can control is whether a fix exists for the things I run when one of them finally matters. On 2026-10-31 a lot of it would not have. Now almost all of it does.