Grafana Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.
How do I use it?
Loki is a central piece of what’s known as the PLG Stack for application monitoring - Promtail, Loki and Grafana. I’m sure a lot of you are already familiar with Grafana data analysis and visualization tools either from the great community add-on or use in some other aspect of your life.
But Grafana is also central to system monitoring. The same company also owns Loki and Promtail which are used to collect and aggregate logs and other metrics from your systems. Then Grafana can pull in this information from Loki so you can explore, analyze, and create metrics and alerts. Grafana isn’t the only tool that can read from Loki but it is usually used in this stack since its all designed to work well together.
Essentially the process you probably want to set up is this:
Promtail scrapes your logs and feeds them to Loki
Loki aggregates and indexes and makes its API available
I found the Loki best practices guide to be very helpful. Has some good advice on how to set the stack up and how to analyze it if you start to run into issues.
Other then that the readme and documentation cover all the options. If you need help, you can:
Not sure if people were waiting or not but the Promtail add-on works on the current stable version of supervisor now. Joining the beta channel is no longer required to try out these add-ons together.
I’m pretty much wanting to do the same thing, but I’m running into the issue of how I can forward Loki that is on my Raspberry Pi (where Home Assistant is installed on) to Grafana that is on my actual Home Server (Debian 10)? I can’t seem to find documentation that states how to forward, it’s always on localhost. Any idea?
Loki doesn’t actually forward things, other things simply talk to it via its API. Promtail sends it logs to aggregate, Grafana sends it queries to answer.
So if you have Grafana running on another machine then all you have to do is this:
List a port to expose on the host for the ‘HTTP Listen Port’ in the Network configuration options of the Loki add-on
Open Grafana and add a Loki data source. For URL put the hostname or IP address of your HA server and then put the port you listed
Then that’s it. Grafana will send all of its queries over to Loki as you put them in the UI.
Is that what you want or were you hoping to get the logs off of HA and onto another machine so you could explore them without depending on the HA machine? If that’s the case then what you want I believe is install the promtail add-on but self-host Loki elsewhere (probably on your Debian 10 server). Then set up promtail to ship the logs over to the other machine so Grafana and Loki can just do their thing in isolation.
Sorry, didn’t see the last paragraph on your post. So I got the Loki and Promtail working on HA, and forwarding them through the port you stated and it works great on Grafana (which is on server). But if I were to go the route of having only Promtail on HA and Loki on the server (which I already do for the server stuff), how would I edit the config or whatever to make this work? I have the config-loki.yml (on server) open and looking at it, but am kind of confused on what I would need to edit to have Promtail on HA forward to Loki on my server, and pretty much do what you stated in your last paragraph.
So essentially I could follow the same steps for other devices for their logs and forward them to Loki on my server.
So the only mandatory part of the promtail add-ons config is that you tell it where to send the logs. So in your case you basically just put this:
client:
url: <hostname/IP of debian server>:<port for Loki>/loki/api/v1/push
That’s really it. If your server uses SSL there’s options for self-signed certs and such and there’s advanced options for log processing detailed out in the docs of the promtail add-on but they’re all optional. If your server is accessible via HTTP and you just want to send all logs that way then what I showed above is all you have to put in the promtail add-on.
For your other services on other servers if you go into the docker container of the promtail add-on you can pull out the config file the add-on generates and use it (it’ll be at /etc/promtail/config.yaml within the container). Although you might want to check out their docker driver. That’s actually the way they recommend pushing logs to Loki from a docker cluster it just wasn’t an option for a supervised HA cluster since it requires a docker plugin.
Oh… I had it almost right from some docs on the internet and from my experimenting. In the server config under the Client, it has a URL which is the 127.0.0.0 type address, and I was putting that in before (into the Promtail url). And it wasn’t working. But the only issue was instead of the 127 address I needed to actually put the server IP address for it to work. Problem solved.
I’ve set up Promtail, Loki, and Grafana on my HA (OS). All seem to be running fine, but when I try to add Loki as a data source in Grafana with the target url of http://39bd2704-loki:3100 it doesn’t work.
I have left all the other fields as they are.
Grafana simply complains that it is “Unable to connect with Loki. Please check the server logs for more details.”.
