Open Source is Dead and I Killed It: How We Self-Hosted Grafana

"Sometimes someone confesses a sin in order to take credit for it." - John von Neumann

In all the talk about vibecoding CRMs and accelerating new software development, one thing that hasn't been mentioned is how much easier it now is to self-host open source software.

Espresso's systems are critical for our customers, so one of the first things we did as a company was set up OpenTelemetry. We pointed the results to Grafana Cloud so that we can get alerted on any blip that happens.

Given we are modifying and maintaining infra for a bunch of customers, this ended up requiring a lot of high cardinality metrics so that we could see issues on any single warehouse any of our customers had, and we also wanted to maintain enough logging to debug any issue that came up in prod from our algorithms.

As our customer count increased, so did our monitoring costs - this summer we were paying close to $10k a month. After pruning unnecessary telemetry, we got this down to about $7.5k/month without losing anything meaningful. But $7,500 is still real money, and Grafana's infra components happen to be open source, so one of the obvious things to look at was: could we just self-host this for much cheaper?

The result

The answer will not shock you: yes, we can. Relying heavily on Codex, we migrated logs/metrics/traces to self-hosted High Availability Loki, Prometheus & Tempo (+ Kafka) and saved $6.5k/month, all with about 2 days of engineering effort.

We kept using the cloud-hosted dashboards, alerts, synthetics, which were still cheap for us, but we managed to move all the expensive chunks off.

A side benefit of this was that we can do more with metrics now that they're 85% cheaper; we used to avoid adding labels to things because the cardinality would blow up, but when things are cheap, you can monitor more.

Why this changes the math

To get on my soap box a bit, this is obviously a bit of a problem for open source software businesses.

We didn't use k8s for the first several years of this company's existence due to the perceived operational complexity, preferring to use ECS and focus on growing our top line revenue. But in the last 6 months, as coding agents have gotten better, we've started comfortably using EKS and running plenty of software on EKS. It's not perfect, and the coding agents don't get things right the first time, but with a staging environment, the blast radius can be limited pretty easily, even for infrastructure changes. And monitoring in particular is pretty exposed; you obviously need it to be reliable

How I did the migration

So, how did I do the actual migration?

1. The thing I really should have done first is get away from the Grafana Alloy Sidecar pattern I had setup when we first started the company to using the shared collectors we had started moving to. This change made everything far simpler since I was no longer deploying every single service for a collector change after that.
2. Ask Codex to setup Loki/Prometheus/Kafka/Tempo via Terraform and change our collector config to start dual writing all telemetry there
3. Ask Codex to bring all our dashboards/alerts into Terraform and create a folder for duplicates of all of these
   • In the end, I had it write a script that would sync down edits from the UI and then apply any fixes we'd made programmatically so that this could happen without bothering the rest of the team with Terraform managed dashboards
4. Ask Codex to setup monitoring dashboards/alerts for monitoring the infrastructure itself
   • Mostly alerts for resources (CPU, RAM, Disk) getting to uncomfortable levels as well as outage alerts
5. Do some random deployments of the self-hosted services to see if they were configured for zero-downtime deployments, then ask Codex to fix them
   • It turns out they were not configured for this, but Codex happily fixed this
   • Having codex monitor metrics every time I did some deployment or other was helpful for filing off the sharp edges, and also adding alerts for things that had gone wrong after the fact
6. Let things run for a while
7. Ask Codex to review the differences between the cloud and self-hosted logs volumes, metrics values, alerts firing
   • There were a couple of settings to be tuned to match cloud behavior
     • out of order time window
     • max trace size before rejection
     • adding log line truncation in the collector
8. Tune the RAM/CPU settings for the various instances for what felt safe
   • We'll go through and lower these more later, for now it's cheap enough that we're keeping it somewhat high so that we can discover peak without failures
9. Ask Codex to find out how much things were costing and whether there were configuration choices that were costing us more money than necessary
   • There were, a bunch of services were provisioning their own load balancer rather than using shared ones

A bonus: fixing our monitoring

As a bonus, while I was working on this I noticed a bunch of problems to be fixed with our monitoring; we'd set most of it up by hand before LLMs were good, but since nobody was really an expert at PromQL we'd made mistakes and Codex did a great job of flagging them. A decent chunk of the time on this migration was spent just paying attention to edge cases in our monitoring that we'd like to fix and not on the self-hosting itself.

What this means for open source

So, what does this mean for open source software? Open source companies have provided a huge benefit to the software industry, and personally, I feel bad for them. Building a business on top of open source wasn't easy in the best of times; today, charging on top of compute just for hosting seems unsustainable. When your code is free, the value of a managed platform is exactly the cost of recreating it; and now that LLMs are driving that cost to zero that value is evaporating.

It's honestly hard to see a path forward for companies built on top of OSS.

‍