For those interested in sending the HA / HassOS (container) logs to something like a local Graylog/ES server or a remote service like Papertrail, I created an add-on for the great Logspout project.
Because HA, especially HassOS, makes it difficult to do something like this in an other way, this add-on, which uses the Docker log API, may be an easy way to send your logging elsewhere.
The configuration is pretty simple. The following will send all logs for example to a local Graylog server using GELF:
I would like to thank you so much for this add-on. I’ve been using it for several weeks now and I’m so happy to finally have my Hassio logs in Rsyslog!
I combine your add on with Papertrail and it works like a charm!
I don’t know Loki, but I did find this Logspout loki adapter. It does seem to use a push api though. If that could work for you I can try to add it.
A pull-based approach would require more work I guess, so if there is any possibility to push the logs to Loki I think I’d prefer to try that first.
Thanks for your reply @bertbaron! I will try to make some more research as soon as I have time. I don’t have excessive experience with log monitoring so the learning curve for me is a bit steep
Thank you so much for creating this! I’ve been searching for something like this for a long while now.
I still think it’s crazy that HomeAssistant doesn’t natively allows remote log delivery.
I’ve set it up to send logs to my Synology log center for now (might have to install and figure out Greylog at some point in the future). However, I had to jump through some hoops to get the it to accept Synology target for syslog and running into some problems, that I can’t figure out since I’m not familiar with Logspout (and couldn’t find any documentation that would help). I had to create a new intake in Synology Log Center, for BSD formatted messages on TCP port 6514. Since my Synology server doesn’t have a valid SSL cert, I didn’t enable requirement for https on it. Then I had to map it to logserver.mydomain.com in nginx to have a valid SSL cert on it. After that I added it as syslog+tls://logserver.mydomain.com:443 to the addon. Addon starts successfully and accepts my route, but then starts going into a loop logging constantly:
2022/06/13 20:40:04 syslog: write tcp 172.30.33.7:49118->192.168.1.110:443: write: broken pipe
2022/06/13 20:40:04 syslog: reconnecting up to 10 times
2022/06/13 20:40:04 syslog: reconnect successful
Any idea what that might be? 192.168.1.110 is IP of my nginx server. Also, I would love to not send all of the logs through nginx, and prefer to send them over UDP anyways, but I couldn’t find how to setup a route for that. Any help would be greatly appreciated.
Edit: Oh, nevermind, I found it in documentation: syslog+udp://internaladdress:6514 that works great! Thanks again for creating a very useful plugin.
@ronnyandre I just released version 1.3.0 with a Loki adapter added. You can use loki://<loki_host>:3100 as route.
I did find this Promptail addon however, which should also work for Loki (there is also a Loki addon by the way which I used for testing). The promptail addon might work better for you (or not). It has at least one advantage: is doesn’t require protection mode to be disabled because it mounts the journald logs.
Even when you decide to use the promptail addon however I would still appreciate it if you would give this adapter a try and give some feedback.
Is there any way to pre-process messages that are collected from docker containers before feeding them into logspout? All stack traces from Supervisor are multiline messages and logspout picks and ships each line as a separate log entry. Would be awesome to aggregate them based on presence of date in the beginning of the message. It’s probably outside of the capabilities of this add-on, but, unfortunately Graylog doesn’t have capabilities of aggregating incoming messages.
Logspout does have some multiline options which I haven’t tried myself. Did you try these? If you manage to get it working it would be nice if you could share your config because others might be interested also.
If that doesn’t work (or work well enough) I’m open to suggestions on how to configure something like that. Actually I’m already playing with the idea to patch logspout on the addon with some mechanism to extract extra fields (like a sensible log level). I do this in Graylog pipelines but it would be nice to be able to do this with the addon and maybe ship some default configuration for commonly used addons so others can benefit from this also without the need to figure this out themselves for their tools.
But these are just thoughts right now, I don’t know how difficult it will be to add this to the code. And what tool or library I could possibly hook in to help with the configuration and execution of such rules.
I’ve tried adding multiline and it works much better then without it, at least all stack traces are aggregated into a single message, but the error itself still shows up as a separate message. I’ll play around with some more options and see if it helps, but at least it’s a significant progress comparing with the default options.
If you can give a hint on how to trigger an exception in the supervisor I might be able to test it myself tomorrow.
It shouldn’t be too difficult to match the supervisor logs correctly I think but the challenge might be to not interfere with the logging of other containers, since the settings are global and each add-on logs in its own unique way. But if we can find good settings I can document those and/or set them as default.
One way I found to reproduce this is to create a template sensor in configuration and use a non-existing entity as one of the items in the value template. Error is reported as two separate lines. First one includes the entire value template, and second one will only say float got invalid input 'unavailable' when rendering template.
Something like this:
But this may incorrectly combine lines for other containers logging in a different way. You can try to make an expression with an or-condition for every container and hope they don’t interfere. The following seems to work for most of mine for example:
Be sure to enter the complete expression on a single line in the config editor, the UI editor has the tendency to break it into multiple lines but this give issues with whitespace characters.
Unless you’re brave enough to try something like that you’ll probably have to stick with what you have now until I’ll provide a better solution, I’m thinking about making this configurable per container but it will take some time to implement this.
Your pattern seems to be working great for all of my containers (as far as I can tell), but I seems to be getting 2 messages for every event now. Not sure why. Any ideas?
I haven’t used logspout before but here’s how I did multiline in the promtail addon linked above:
Both supervisor and HA start every log line with the Unicode character to tell the terminal what color to use. I found that was the easiest way to find line start rather then trying to pattern match the date personally.
The problem with date matching is not all log lines actually start with a date, only the ones output by the python code do. The ones output during setup and breakdown of s6 don’t and then get messed up. Which isn’t that big a deal but the color code char seemed simpler and I found it always works.
And btw, easiest way to get supervisor to have an exception is to add a bad addon repository. Just add google.com or something which obviously isn’t a repo.
Thanks, that’s a nice trick. Unfortunately Logspout’s default multiline adapter doesn’t allow configuration per container. But I’m thinking about creating my own adapter with support for container-specific configuration, in which case your example might be useful.
Ah hm, that is unfortunate. Well if I could only pick one multiline option I’d probably pick matching on that unicode character then for sure because:
Basically every python container in the HA ecosystem uses it. Core, Supervisor, even the Google Drive Backup add-on will all have multiline correctly parsed with that
Every non-python container in the HA ecosystem will simply ignore it I believe. Depends a bit on how logspout handles match misses but none of the non-python containers I’ve seen output the terminal color-coding characters so they should just behave normally.
Other patterns like date matching mostly work for core and supervisor (minus the s6 stuff) but likely run into a greater chance of accidental matches leading to incorrect parses in other addons and plugins. Plus its a more expensive pattern in general in terms of regex processing.
Anyway just my 2 cents, interesting project. I have a much larger list of promtail stages I use for parsing the logs of my own addons correctly to get things like multi-line working in the various log formats. Feel free to ping me if you’re interested. Here, PM on the forum, I’m also CentralCommand#0913 in the HA discord.
However, now that it’s been running for a little bit, I think it might be skipping messages from some containers. For example I don’t see any messages from either NodeRed or Zigbee2mqtt since I added multiline pattern in. Zwave2mqtt logs are still there, but they are also duplicated, same as Home Assistant Core logs.
And yes, I can confirm that the patter above doesn’t work for all containers. Mosquitto logs are getting over aggregated, since they don’t start with the timestamp.
