Today I discovered that a container in my home Kubernetes crashed, which is responsible for downloading the price of the SP500 ETF fund, which is then taken by Prometheus and passed to my Grafana.

I went through the logs a bit, and realized that it’s probably due to using an old Python library, because as scraping changes, new libraries are released that adapt to the website. However, my docker image published on GitHub was last updated about a year ago, and is now outdated. I thought it could be automatically built every month with the latest version of the library, which can be done in GitHub actions like this:

name: CNB Exporter PROD CI

on:  
  workflow_dispatch: # Adds manual trigger
  schedule:
  # Runs at 05:05 UTC on the 5th day of every month
  # ┌───────────── minute (0 - 59)
  # │ ┌───────────── hour (0 - 23)
  # │ │ ┌───────────── day of the month (1 - 31)
  # │ │ │ ┌───────────── month (1 - 12)
  # │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
  # │ │ │ │ │
  # │ │ │ │ │
  # * * * * *
  - cron: '5 5 9 * *'

I played with it a bit more and set everything to use my local Runner, which runs on my server, and can also restart the deployment on Kubernetes, which when set to Pull: Always automatically downloads the new image. It’s not rocket science, you just need to set up the Runner to have its own role in Kubernetes, which has the right to restart deployments, the role looks something like this:

kind: Role
...
rules:
- apiGroups:
  - apps
  resourceNames:
  - '*'
  resources:
  - deployments
  verbs:
  - get
  - list
  - patch

And suddenly everything works again. I wouldn’t recommend deploying the latest version without testing in a corporate environment, I would probably first write some tests to test it, but at home, from the perspective that I don’t completely depend on it and time possibilities are limited, I won’t dedicate time to this.

And this is what the outage looked like in Grafana#

Grafana