Grafana Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.
How do I use it?
Loki is a central piece of what’s known as the PLG Stack for application monitoring - Promtail, Loki and Grafana. I’m sure a lot of you are already familiar with Grafana data analysis and visualization tools either from the great community add-on or use in some other aspect of your life.
But Grafana is also central to system monitoring. The same company also owns Loki and Promtail which are used to collect and aggregate logs and other metrics from your systems. Then Grafana can pull in this information from Loki so you can explore, analyze, and create metrics and alerts. Grafana isn’t the only tool that can read from Loki but it is usually used in this stack since its all designed to work well together.
Essentially the process you probably want to set up is this:
Promtail scrapes your logs and feeds them to Loki
Loki aggregates and indexes and makes its API available
I found the Loki best practices guide to be very helpful. Has some good advice on how to set the stack up and how to analyze it if you start to run into issues.
Other then that the readme and documentation cover all the options. If you need help, you can:
Not sure if people were waiting or not but the Promtail add-on works on the current stable version of supervisor now. Joining the beta channel is no longer required to try out these add-ons together.
I’m pretty much wanting to do the same thing, but I’m running into the issue of how I can forward Loki that is on my Raspberry Pi (where Home Assistant is installed on) to Grafana that is on my actual Home Server (Debian 10)? I can’t seem to find documentation that states how to forward, it’s always on localhost. Any idea?
Loki doesn’t actually forward things, other things simply talk to it via its API. Promtail sends it logs to aggregate, Grafana sends it queries to answer.
So if you have Grafana running on another machine then all you have to do is this:
List a port to expose on the host for the ‘HTTP Listen Port’ in the Network configuration options of the Loki add-on
Open Grafana and add a Loki data source. For URL put the hostname or IP address of your HA server and then put the port you listed
Then that’s it. Grafana will send all of its queries over to Loki as you put them in the UI.
Is that what you want or were you hoping to get the logs off of HA and onto another machine so you could explore them without depending on the HA machine? If that’s the case then what you want I believe is install the promtail add-on but self-host Loki elsewhere (probably on your Debian 10 server). Then set up promtail to ship the logs over to the other machine so Grafana and Loki can just do their thing in isolation.
Sorry, didn’t see the last paragraph on your post. So I got the Loki and Promtail working on HA, and forwarding them through the port you stated and it works great on Grafana (which is on server). But if I were to go the route of having only Promtail on HA and Loki on the server (which I already do for the server stuff), how would I edit the config or whatever to make this work? I have the config-loki.yml (on server) open and looking at it, but am kind of confused on what I would need to edit to have Promtail on HA forward to Loki on my server, and pretty much do what you stated in your last paragraph.
So essentially I could follow the same steps for other devices for their logs and forward them to Loki on my server.
So the only mandatory part of the promtail add-ons config is that you tell it where to send the logs. So in your case you basically just put this:
client:
url: <hostname/IP of debian server>:<port for Loki>/loki/api/v1/push
That’s really it. If your server uses SSL there’s options for self-signed certs and such and there’s advanced options for log processing detailed out in the docs of the promtail add-on but they’re all optional. If your server is accessible via HTTP and you just want to send all logs that way then what I showed above is all you have to put in the promtail add-on.
For your other services on other servers if you go into the docker container of the promtail add-on you can pull out the config file the add-on generates and use it (it’ll be at /etc/promtail/config.yaml within the container). Although you might want to check out their docker driver. That’s actually the way they recommend pushing logs to Loki from a docker cluster it just wasn’t an option for a supervised HA cluster since it requires a docker plugin.
Oh… I had it almost right from some docs on the internet and from my experimenting. In the server config under the Client, it has a URL which is the 127.0.0.0 type address, and I was putting that in before (into the Promtail url). And it wasn’t working. But the only issue was instead of the 127 address I needed to actually put the server IP address for it to work. Problem solved.
I’ve set up Promtail, Loki, and Grafana on my HA (OS). All seem to be running fine, but when I try to add Loki as a data source in Grafana with the target url of http://39bd2704-loki:3100 it doesn’t work.
I have left all the other fields as they are.
Grafana simply complains that it is “Unable to connect with Loki. Please check the server logs for more details.”.
Is this addon still live / functioning? Just tried to install, and get the following error…
-----------------------------------------------------------
Add-on: Loki
Loki for Home Assistant
-----------------------------------------------------------
Add-on version: 1.11.2
You are running the latest version of this add-on.
System: Home Assistant OS 14.1 (aarch64 / yellow)
Home Assistant Core: 2024.12.5
Home Assistant Supervisor: 2024.12.0
-----------------------------------------------------------
Please, share the above information when looking for help
or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
cont-init: info: /etc/cont-init.d/00-banner.sh exited 0
cont-init: info: running /etc/cont-init.d/01-log-level.sh
Log level is set to INFO
cont-init: info: /etc/cont-init.d/01-log-level.sh exited 0
cont-init: info: running /etc/cont-init.d/nginx.sh
cont-init: info: /etc/cont-init.d/nginx.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun loki (no readiness notification)
services-up: info: copying legacy longrun nginx (no readiness notification)
s6-rc: info: service legacy-services successfully started
[10:50:27] INFO: Starting Loki...
[10:50:27] INFO: Using default config
[10:50:27] INFO: Retention period set to 30d
[10:50:27] INFO: Loki log level set to info
[10:50:27] INFO: Handing over control to Loki...
level=info ts=2024-12-31T15:50:28.113357413Z caller=main.go:103 msg="Starting Loki" version="(version=2.6.1, branch=HEAD, revision=6bd05c9a4)"
level=info ts=2024-12-31T15:50:28.114493665Z caller=server.go:288 http=127.0.0.1:8080 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2024-12-31T15:50:28.115251864Z caller=modules.go:736 msg="RulerStorage is not configured in single binary mode and will not be started."
level=warn ts=2024-12-31T15:50:28.116802576Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksfifocache"
mkdir /data/loki/chunks: permission denied
error creating object client
github.com/grafana/loki/pkg/storage.(*store).chunkClientForPeriod
/src/loki/pkg/storage/store.go:185
github.com/grafana/loki/pkg/storage.(*store).init
/src/loki/pkg/storage/store.go:153
github.com/grafana/loki/pkg/storage.NewStore
/src/loki/pkg/storage/store.go:145
github.com/grafana/loki/pkg/loki.(*Loki).initStore
/src/loki/pkg/loki/modules.go:522
github.com/grafana/dskit/modules.(*Manager).initModule
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92
github.com/grafana/loki/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:341
main.main
/src/loki/cmd/loki/main.go:105
runtime.main
/usr/local/go/src/runtime/proc.go:255
runtime.goexit
/usr/local/go/src/runtime/asm_arm64.s:1133
error initialising module: store
github.com/grafana/dskit/modules.(*Manager).initModule
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92
github.com/grafana/loki/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:341
main.main
/src/loki/cmd/loki/main.go:105
runtime.main
/usr/local/go/src/runtime/proc.go:255
runtime.goexit
/usr/local/go/src/runtime/asm_arm64.s:1133
level=error ts=2024-12-31T15:50:28.117906717Z caller=log.go:103 msg="error running loki" err="mkdir /data/loki/chunks: permission denied\nerror creating object client\ngithub.com/grafana/loki/pkg/storage.(*store).chunkClientForPeriod\n\t/src/loki/pkg/storage/store.go:185\ngithub.com/grafana/loki/pkg/storage.(*store).init\n\t/src/loki/pkg/storage/store.go:153\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:145\ngithub.com/grafana/loki/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:522\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:341\nmain.main\n\t/src/loki/cmd/loki/main.go:105\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1133\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:341\nmain.main\n\t/src/loki/cmd/loki/main.go:105\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1133"
[15:50:28] WARNING: Halt add-on with exit code 1
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
