Setting up Grafana, Loki & Promtail

Community Guides
1

Introduction
I’ve spent a few hours over the last few days getting set up with Grafana, Loki & Promtail. Most people will know Grafana but Loki & Promtail basically allow you to centralise your system logs and they also allow you to see said logs in Grafana. As there were a few ‘bumps’ along the road for me, I thought I’d write a guide for helping anyone else who may attempt this feat!

Warning
Before I start, a word of warning - I’m in no way an expert and had to do a lot of Googling (some lots of guessing) to get everything set up working for me. So follow this guide at your own peril!

My setup
HP Microserver Gen8 running Ubuntu 18.04.5. I have Home Assistant set up inside a Docker container on this machine and I also wanted all of this to be in Docker containers too (more on this later).

Grafana
To set up Grafana inside a Docker container I used this guide on the forum, however it is reasonably old and omits what I found to be a major step - It doesn’t make the data persistent so when you spin down your Grafana container, the data is lost (I found this out the hard way)! So here is the command I used to get my container set up:

docker create \
  --name="grafana" \
  -e PUID=996 \
  -e PGID=995 \
  -p 3000:3000 \
  -v /opt/docker/grafana/grafana.ini:/etc/grafana/grafana.ini \
  -v /opt/docker/grafana/data:/var/lib/grafana \
  -v /opt/docker/grafana/logs:/var/log/grafana \
  -e "GF_LOG_MODE=console file" \
  --restart unless-stopped \
  grafana/grafana

You will need to change the PUID, GUID and volume locations for your own setup.
More information from Grafana themselves can be found here and, with regards to the persistent storage, here.

Loki
To set up the Loki container I followed this guide on the Grafana website.
Note: The linked guide show steps for Promtail and Loki, after trial and error I only used the Loki aspect of this guide - I will explain why within the Promtail section.
The command I used to set up the Loki container:

docker create \
  --name="loki" \
  -e PUID=995 \
  -e PGID=994 \
  -p 3100:3100 \
  -v /opt/docker/loki:/mnt/config \
  --restart unless-stopped \
  grafana/loki:2.0.0 -config.file=/mnt/config/loki-config.yaml

Again, you will need to change the PUID, GUID and volume locations for your own setup.

Promtail
You can set up Promtail in a Docker container, however I quickly found that it became very tedious as in order to ingest each log file you also have to whitelist it as a volume when creating the Docker container. There is perhaps an easier way to do it, but I wasn’t aware and I couldn’t spot anything on Google so decided to install it using the Binary instead (ie. not within a container). It is actually really simple as it’s just one program file, one configuration file, a service file and a few commands to get it all set up.
Luckily, there’s a guide online alerady for setting up Promtail as a Binary, so I just followed that! You can find it here.

Bringing it all together
Within the linked guides, there are checks you can do to make sure that Loki and Promtail are talking to each other. There are multiple guides on how to ingest different logs into Promtail, but just to show one example - This is how I have my HA logs fed into Promtail:

- job_name: homeassistant
  static_configs:
  - targets:
      - localhost
    labels:
      job: homeassistant
      __path__: /home/zuluwhiskey/homeassistant/home-assistant.log

To then get Loki into Grafana, just follow this quick guide!

So now, I can annotate my graphs with Home Assistant restarts :smiley:

grafannoCapture
grafannoCapture1271×440 27.9 KB

hagrafCapture

As I said at the beginning, I’m no expert but I’ll be happy to try and answer any questions people may have! :+1:

2 Likes
2

Just an FYI for those interested in this topic but running Supervisor (Home Assistant OS or Supervised install), I created add-ons for Loki and Promtail. And obviously the Grafana add-on has been in the Community add-ons repository for some time.

While not impossible to start up docker containers directly in a supervised deployment, its difficult and not really recommended. So for those in that situation, these addons provide an easy alternative.

Also @ZuluWhiskey, based on your instructions I’m guessing you’re running a Home Assistant Container setup? If so you should take a look at the Promtail Docker Logging Driver. That’s their recommended approach for shipping logs to Loki in a Docker setup. It seems like it makes it really easy to ship all logs from all containers right to Loki, HA included.

I wanted to use the driver but unfortunately you can’t in a supervised deployment, journal logging is the only supported logging driver (and it broke pretty spectacularly when I tried to force it in anyway :slight_smile: ). But now that add-ons can read from the system journal, the add-on solves the problem for those of us with less control over our docker deployment.

2 Likes
3

Apologies for the very late reply, I somehow missed your comment!

Nice work with the Loki & Promtail add-ons :+1:

My HA is indeed running in a container! I’ll take a look at the logging driver, thanks for the heads up as I definitely wasn’t aware of that.

I’ve also just noticed that all of my guide is using docker run commands :face_vomiting:. I recently moved to docker-compose and it has been a life saver so will take the time to update the above at some point with a compose file!

2 Likes