Logspout add-on for sending HA logs to log management systems

Share your Projects!
#21

Nope, just one route. Below is my configuration yaml:

routes:
  - multiline+gelf://storageserver.home:12201
env:
  - name: SYSLOG_HOSTNAME
    value: homeassistant
  - name: INACTIVITY_TIMEOUT
    value: 1m
  - name: MULTILINE_PATTERN
    value: >-
      (\d\d(\d\d)?[-/]\d\d[-/]\d\d[T
      ]\d\d:\d\d:\d\d)|(^s6-rc:)|(^\[\d\d:\d\d:\d\d\])|(\d\d:\d\d:\d\d\
      -)|(^[TDIWEF]:)
  - name: MULTILINE_MATCH
    value: first

However, now that it’s been running for a little bit, I think it might be skipping messages from some containers. For example I don’t see any messages from either NodeRed or Zigbee2mqtt since I added multiline pattern in. Zwave2mqtt logs are still there, but they are also duplicated, same as Home Assistant Core logs.

And yes, I can confirm that the patter above doesn’t work for all containers. Mosquitto logs are getting over aggregated, since they don’t start with the timestamp.

#22

I guess it’s just too tricky to get the default multiline adapter to work. Instead of trying I better spend time on a proper adapter that works well for HA and comes preconfigured for at least the core containers. I hope you can be patient, it might take a few weeks or a little more but I promise it will come.

#23

Sounds good. I’ve waited for years to be able to get HA logs out, so waiting a few more weeks isn’t really an issue.
One thing to note, even once I removed all of the multiline patterns from the config, I’m still getting duplicated messages for all containers. And I think it started after the latest update to the addon. Is it possible that something changed in the addon itself that duplicates messages?

#24

I can’t think of anything. Do you also get duplicate lines without using multiline on your route? Maybe it’s a bug in that adaper.
Other thinks you might want to look at:

  • Does the log from logspout show anything weird
  • Check the configuration as yaml (the GUI editor fooled me more than once)
  • Set the Debug variable:
    - name: DEBUG
  value: "true"
    for me it doesn’t show much extra but it does show 16 pump.getLogs() messages and I have 16 containers running

If nothing helps I would like to know if exactly each message is duplicated or if it seems to be more random.

#25

Yea they’re each different unfortunately. Here’s what I use for Z2M and nodered in my promtail config:

- match:
    selector: '{container_name="addon_45df7312_zigbee2mqtt"}'
    stages:
      - multiline:
          firstline: '^(\x{001b}|\(node:\d+\)|\[[-_.:a-z0-9]\]) '
- match:
    selector: '{container_name="addon_a0d7b954_nodered"}'
    stages:
      - multiline:
          firstline: '^(\d{1,2} [A-Za-z]{3,4} \d\d(?::\d\d){2} -|\[[-_.a-z0-9]+\]) '

Also just an FYI, Z2M actually logs to a file in its config folder by default (probably either /config/zigbee2mqtt or /share/zigbee2mqtt depending on when you set it up). Once you have an easy way to review the docker logs for addons you should disable that so it stops chewing up extra disk cycles. You can do this by changing “Log Output” to console and blanking out the “Log File” and “Log Directory” fields in advanced settings.

I don’t use Zwave at all so I don’t know that one. That’s pretty weird though, why would they be duplicated?

I actually didn’t find any need to do multiline for mosquitto. The only thing I have for that one is this:

- match:
    selector: '{container_name="addon_core_mosquitto"}'
    stages:
      - drop:
          expression: ".*Saving in-memory database.*"
          drop_counter_reason: Cron

It pops that line in the log all the time so I just drop it. I don’t have to review the logs of this addon much but its not really helpful to have pages and pages of that message in grafana when I do.

EDIT: I just realized looking at my post that Z2M also uses the terminal color code character. So it looks like that pattern mostly works for that addon as well. If memory serves its a bit more then that there because of exceptions. I believe uncaught exceptions just start with node: and then dump a stack trace. Something like that.