software

As part of my work life in the past year, a chunk of my day-to-day life has consisted of maintaining a prometheus installation on top of a sizable kubernetes cluster. My original feeling was “this is not that bad with kube-prometheus-stack”, but this sentiment has worsened somewhat with the realisation that more and more customizations and pieces were needed for large scale use. Half a year later (and 6+ charts deep), I thought I’d collect my thoughts on the ecosystem - from an operational perspective - with a rough architecture overview post.

First KubeCon in a while I haven’t done anything for (didn’t even buy an ticket). This post is largely for myself, but thought I’d put some thoughts here public. All talks referenced were recently published on the CNCF youtube channel, and the posts here are really just my notes (make of them what you will).

After a quarter year of extensive improvements to kube, it’s time to take a birds-eye view of what we got, and showcase some of the recent improvements. After all, it’s been about 40 kube releases, one major version of tokio, one extremely prolific new contributor, and one kubecon talk since my (very outdated) last blog post.

Despite having been a lazy Cinnamon customer since the Linux Mint days, recently some interesting benchmarks started surfacing about input lag in various Window Mangagers, and this made me want to experiment a little.

This is a log of stuff I needed to tweak to get LXDE working well.

It’s been about a month since we released kube, a new rust client library for kubernetes. We covered the initial release, but it was full of naive optimism and uncertainty. Would the generic setup work with native objects? How far would it extend? Non-standard objects? Patch handling? Event handling? Surely, it’d be a fools errand to write an entire client library?

With the last 0.10.0 release, it’s now clear that the generic setup extends quite far. Unfortunately, this yak is hairy, even by yak standards.

When interacting with kubernetes it’s generally been standard practice to use either client-go via go, or kubectl via shell.

While these are good, non-controversial choices, the advancement of client libraries, and smarter openapi bindings, combined with the generics and procedural macros of rust-lang, it’s now quite possible to write fully fledged kube operators, using slim rust kube clients.

Authenticating with large kubernetes clusters often risks you dealing with complicated provider logic and sometimes policies outside your control.

While controllers and operators authenticate with service accounts directly, this is only true inside the cluster. That is, unless you can impersonate the service account from outside.

At babylon health we have a ton of microservices running on kubernetes that are, in turn, controlled by hundreds of thousands of lines of autogenerated yaml.

So for our own sanity, we built shipcat - a standardisation tool (powered by rust-lang and serde) to control the declarative format and lifecycle of every microservice.

A particularly tricky problem with tournament scheduling that made it into tournament/duel.