Homeassistant + prometheus on k8s: Login attempt or request with invalid authentication

Installation Home Assistant OS
1

I’m running both Homeassistant and Prometheus on a kubernetes cluster. I have setup a long lived access token and enabled the prometheus integration in the config file (prometheus:) If I execute curl with this token, I get the expected outcome;

curl -X GET -H “Authorization: Bearer [my_token]” -H “Content-Type: application/json” https://my_server]/api/prometheus

I can’t paste my entire serviceMonitor yaml because of forum restrictions, but here are the relevant parts;

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    release: prometheus
spec:
  endpoints:
  - bearerTokenSecret:
      key: [my_token]
    interval: 30s

I can see the target in Prometheus, but it is DOWN with the error ‘server returned HTTP status 401 Unauthorized’. In Homeassistant, I see a ‘Login attempt or request with invalid authentication’ every 30s.

What did I do wrong?

2

I guess you’ll need to read up on how to store and retrieve secrets in k8s. The key field value should correspond with a stored secret name in k8s secrets store.

I personally find k8s a huge overkill, I run my set up in a Hashicorp Nomad cluster with prometheus etcetera. Metrics and such work perfectly with the use of a token. This should also work perfectly in k8s. But have a look at https://kubernetes.io/docs/concepts/configuration/secret/ for reading up on how to use secrets in k8s.

3

You were right, I thought it was the secret referred to the long lived token. I found another good source explaining this: https://docs.couchbase.com/operator/2.0/tutorial-prometheus.html

k8s is huge overkill, but I want to gain some knowledge as we are also using it at work.

Thanks for the help, I appreciate it!

secret.yaml;

apiVersion: v1
kind: Secret
metadata:
  name: prometheus-hass
type: Opaque
stringData:
  token: <long-lived access token>

servicemonitor.yaml;

spec:
  endpoints:
  - bearerTokenSecret:
      key: token
      name: prometheus-hass