I’ve tried adding multiline and it works much better then without it, at least all stack traces are aggregated into a single message, but the error itself still shows up as a separate message. I’ll play around with some more options and see if it helps, but at least it’s a significant progress comparing with the default options.
If you can give a hint on how to trigger an exception in the supervisor I might be able to test it myself tomorrow.
It shouldn’t be too difficult to match the supervisor logs correctly I think but the challenge might be to not interfere with the logging of other containers, since the settings are global and each add-on logs in its own unique way. But if we can find good settings I can document those and/or set them as default.
One way I found to reproduce this is to create a template sensor in configuration and use a non-existing entity as one of the items in the value template. Error is reported as two separate lines. First one includes the entire value template, and second one will only say float got invalid input 'unavailable' when rendering template.
Something like this:
- platform: template
sensors:
some_sensor:
value_template: "{{ states(sensor.some_entity') | float - states(sensor.some_other_entity) | float - state(sensor.undefined_entity) | float }}"
Specifically for the home assistant container these seem to work
- name: MULTILINE_PATTERN
value: \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}
- name: MULTILINE_MATCH
value: first
But this may incorrectly combine lines for other containers logging in a different way. You can try to make an expression with an or-condition for every container and hope they don’t interfere. The following seems to work for most of mine for example:
(\d\d(\d\d)?[-/]\d\d[-/]\d\d[T ]\d\d:\d\d:\d\d)|(^s6-rc:)|(^\[\d\d:\d\d:\d\d\])|(\d\d:\d\d:\d\d\ -)|(^[TDIWEF]:)
Be sure to enter the complete expression on a single line in the config editor, the UI editor has the tendency to break it into multiple lines but this give issues with whitespace characters.
Unless you’re brave enough to try something like that you’ll probably have to stick with what you have now until I’ll provide a better solution, I’m thinking about making this configurable per container but it will take some time to implement this.
Your pattern seems to be working great for all of my containers (as far as I can tell), but I seems to be getting 2 messages for every event now. Not sure why. Any ideas?
Check out this one instead.
You didn’t accidentally define your route twice? If not, then I don’t know. If you share your add-on configuration (as yaml) I’ll have a look.
I haven’t used logspout before but here’s how I did multiline in the promtail addon linked above:
Both supervisor and HA start every log line with the Unicode character to tell the terminal what color to use. I found that was the easiest way to find line start rather then trying to pattern match the date personally.
The problem with date matching is not all log lines actually start with a date, only the ones output by the python code do. The ones output during setup and breakdown of s6 don’t and then get messed up. Which isn’t that big a deal but the color code char seemed simpler and I found it always works.
And btw, easiest way to get supervisor to have an exception is to add a bad addon repository. Just add google.com or something which obviously isn’t a repo.
Thanks, that’s a nice trick. Unfortunately Logspout’s default multiline adapter doesn’t allow configuration per container. But I’m thinking about creating my own adapter with support for container-specific configuration, in which case your example might be useful.
Ah hm, that is unfortunate. Well if I could only pick one multiline option I’d probably pick matching on that unicode character then for sure because:
- Basically every python container in the HA ecosystem uses it. Core, Supervisor, even the Google Drive Backup add-on will all have multiline correctly parsed with that
- Every non-python container in the HA ecosystem will simply ignore it I believe. Depends a bit on how logspout handles match misses but none of the non-python containers I’ve seen output the terminal color-coding characters so they should just behave normally.
- Other patterns like date matching mostly work for core and supervisor (minus the s6 stuff) but likely run into a greater chance of accidental matches leading to incorrect parses in other addons and plugins. Plus its a more expensive pattern in general in terms of regex processing.
Anyway just my 2 cents, interesting project. I have a much larger list of promtail stages I use for parsing the logs of my own addons correctly to get things like multi-line working in the various log formats. Feel free to ping me if you’re interested. Here, PM on the forum, I’m also CentralCommand#0913 in the HA discord.
Nope, just one route. Below is my configuration yaml:
routes:
- multiline+gelf://storageserver.home:12201
env:
- name: SYSLOG_HOSTNAME
value: homeassistant
- name: INACTIVITY_TIMEOUT
value: 1m
- name: MULTILINE_PATTERN
value: >-
(\d\d(\d\d)?[-/]\d\d[-/]\d\d[T
]\d\d:\d\d:\d\d)|(^s6-rc:)|(^\[\d\d:\d\d:\d\d\])|(\d\d:\d\d:\d\d\
-)|(^[TDIWEF]:)
- name: MULTILINE_MATCH
value: first
However, now that it’s been running for a little bit, I think it might be skipping messages from some containers. For example I don’t see any messages from either NodeRed or Zigbee2mqtt since I added multiline pattern in. Zwave2mqtt logs are still there, but they are also duplicated, same as Home Assistant Core logs.
And yes, I can confirm that the patter above doesn’t work for all containers. Mosquitto logs are getting over aggregated, since they don’t start with the timestamp.
I guess it’s just too tricky to get the default multiline adapter to work. Instead of trying I better spend time on a proper adapter that works well for HA and comes preconfigured for at least the core containers. I hope you can be patient, it might take a few weeks or a little more but I promise it will come.
Sounds good. I’ve waited for years to be able to get HA logs out, so waiting a few more weeks isn’t really an issue.
One thing to note, even once I removed all of the multiline patterns from the config, I’m still getting duplicated messages for all containers. And I think it started after the latest update to the addon. Is it possible that something changed in the addon itself that duplicates messages?
I can’t think of anything. Do you also get duplicate lines without using multiline on your route? Maybe it’s a bug in that adaper.
Other things you might want to look at:
- Does the log from logspout show anything weird
- Check the configuration as yaml (the GUI editor fooled me more than once)
- Set the Debug variable:
for me it doesn’t show much extra but it does show 16- name: DEBUG value: "true"pump.getLogs()messages and I have 16 containers running
If nothing helps I would like to know if exactly each message is duplicated or if it seems to be more random.
Yea they’re each different unfortunately. Here’s what I use for Z2M and nodered in my promtail config:
- match:
selector: '{container_name="addon_45df7312_zigbee2mqtt"}'
stages:
- multiline:
firstline: '^(\x{001b}|\(node:\d+\)|\[[-_.:a-z0-9]\]) '
- match:
selector: '{container_name="addon_a0d7b954_nodered"}'
stages:
- multiline:
firstline: '^(\d{1,2} [A-Za-z]{3,4} \d\d(?::\d\d){2} -|\[[-_.a-z0-9]+\]) '
Also just an FYI, Z2M actually logs to a file in its config folder by default (probably either /config/zigbee2mqtt or /share/zigbee2mqtt depending on when you set it up). Once you have an easy way to review the docker logs for addons you should disable that so it stops chewing up extra disk cycles. You can do this by changing “Log Output” to console and blanking out the “Log File” and “Log Directory” fields in advanced settings.
I don’t use Zwave at all so I don’t know that one. That’s pretty weird though, why would they be duplicated?
I actually didn’t find any need to do multiline for mosquitto. The only thing I have for that one is this:
- match:
selector: '{container_name="addon_core_mosquitto"}'
stages:
- drop:
expression: ".*Saving in-memory database.*"
drop_counter_reason: Cron
It pops that line in the log all the time so I just drop it. I don’t have to review the logs of this addon much but its not really helpful to have pages and pages of that message in grafana when I do.
EDIT: I just realized looking at my post that Z2M also uses the terminal color code character. So it looks like that pattern mostly works for that addon as well. If memory serves its a bit more then that there because of exceptions. I believe uncaught exceptions just start with node: and then dump a stack trace. Something like that.
I also want to suppress the addon_core_mosquitto messages. Can you give me a hint where to put this match code you shared ?
The match code form @CentralCommand refers to the configuration of promptail. Logspout does not have such a configuration. I still have it on my todo list to add something like that in a simplified form but didn’t find the time yet.
Logspout can filter logs based on container labels (GitHub - gliderlabs/logspout: Log routing for Docker container logs). Unfortunately with HA we don’t have much control over those, but fortunately home assistant does assign some labels by itself, in particular the label io.has.name might be useful. In your case you might try:
env:
- name: EXCLUDE_LABELS
value: io.hass.name:Mosquitto broker
in your configuration. I didn’t fully test this myself yet, but please let us know if it works.
1 Like
Thank you, Bert.
Your suggestion works perfectly!
Can you also think of a way to exclude log entries from hassio_dns ?
If you have access to the cli somehow you can try to run something like this to find the labels:
# docker inspect hassio_dns | jq " .[0].Config.Labels"
{
"io.hass.arch": "amd64",
"io.hass.base.arch": "amd64",
"io.hass.base.image": "alpine:3.14",
"io.hass.base.name": "alpine",
"io.hass.base.version": "2022.02.0",
"io.hass.type": "dns",
"io.hass.version": "2022.04.1",
"org.opencontainers.image.authors": "The Home Assistant Authors",
"org.opencontainers.image.created": "2022-04-26 14:37:01+00:00",
"org.opencontainers.image.description": "Home Assistant Supervisor plugin for DNS",
"org.opencontainers.image.documentation": "https://www.home-assistant.io/docs/",
"org.opencontainers.image.licenses": "Apache License 2.0",
"org.opencontainers.image.source": "https://github.com/home-assistant/plugin-dns",
"org.opencontainers.image.title": "Home Assistant DNS Plugin",
"org.opencontainers.image.url": "https://www.home-assistant.io/",
"org.opencontainers.image.version": "2022.04.1",
"supervisor_managed": ""
}
The most useful label seems to be io.hass.type. So your configuration should look like:
env:
- name: EXCLUDE_LABELS
value: io.hass.name:Mosquitto broker;io.hass.type:dns
I hope this works.
2 Likes