I’ve spent a few hours over the last few days getting set up with Grafana, Loki & Promtail. Most people will know Grafana but Loki & Promtail basically allow you to centralise your system logs and they also allow you to see said logs in Grafana. As there were a few ‘bumps’ along the road for me, I thought I’d write a guide for helping anyone else who may attempt this feat!
Before I start, a word of warning - I’m in no way an expert and had to do a lot of Googling (
some lots of guessing) to get everything set up working for me. So follow this guide at your own peril!
HP Microserver Gen8 running Ubuntu 18.04.5. I have Home Assistant set up inside a Docker container on this machine and I also wanted all of this to be in Docker containers too (more on this later).
To set up Grafana inside a Docker container I used this guide on the forum, however it is reasonably old and omits what I found to be a major step - It doesn’t make the data persistent so when you spin down your Grafana container, the data is lost (I found this out the hard way)! So here is the command I used to get my container set up:
docker create \ --name="grafana" \ -e PUID=996 \ -e PGID=995 \ -p 3000:3000 \ -v /opt/docker/grafana/grafana.ini:/etc/grafana/grafana.ini \ -v /opt/docker/grafana/data:/var/lib/grafana \ -v /opt/docker/grafana/logs:/var/log/grafana \ -e "GF_LOG_MODE=console file" \ --restart unless-stopped \ grafana/grafana
To set up the Loki container I followed this guide on the Grafana website.
Note: The linked guide show steps for Promtail and Loki, after trial and error I only used the Loki aspect of this guide - I will explain why within the Promtail section.
The command I used to set up the Loki container:
docker create \ --name="loki" \ -e PUID=995 \ -e PGID=994 \ -p 3100:3100 \ -v /opt/docker/loki:/mnt/config \ --restart unless-stopped \ grafana/loki:2.0.0 -config.file=/mnt/config/loki-config.yaml
Again, you will need to change the PUID, GUID and volume locations for your own setup.
You can set up Promtail in a Docker container, however I quickly found that it became very tedious as in order to ingest each log file you also have to whitelist it as a volume when creating the Docker container. There is perhaps an easier way to do it, but I wasn’t aware and I couldn’t spot anything on Google so decided to install it using the Binary instead (ie. not within a container). It is actually really simple as it’s just one program file, one configuration file, a service file and a few commands to get it all set up.
Luckily, there’s a guide online alerady for setting up Promtail as a Binary, so I just followed that! You can find it here.
Bringing it all together
Within the linked guides, there are checks you can do to make sure that Loki and Promtail are talking to each other. There are multiple guides on how to ingest different logs into Promtail, but just to show one example - This is how I have my HA logs fed into Promtail:
- job_name: homeassistant static_configs: - targets: - localhost labels: job: homeassistant __path__: /home/zuluwhiskey/homeassistant/home-assistant.log
To then get Loki into Grafana, just follow this quick guide!
So now, I can annotate my graphs with Home Assistant restarts
As I said at the beginning, I’m no expert but I’ll be happy to try and answer any questions people may have!