Logspout add-on for sending HA logs to log management systems

ivolodin · June 22, 2022, 3:24am

Nope, just one route. Below is my configuration yaml:

routes:
  - multiline+gelf://storageserver.home:12201
env:
  - name: SYSLOG_HOSTNAME
    value: homeassistant
  - name: INACTIVITY_TIMEOUT
    value: 1m
  - name: MULTILINE_PATTERN
    value: >-
      (\d\d(\d\d)?[-/]\d\d[-/]\d\d[T
      ]\d\d:\d\d:\d\d)|(^s6-rc:)|(^\[\d\d:\d\d:\d\d\])|(\d\d:\d\d:\d\d\
      -)|(^[TDIWEF]:)
  - name: MULTILINE_MATCH
    value: first

However, now that it’s been running for a little bit, I think it might be skipping messages from some containers. For example I don’t see any messages from either NodeRed or Zigbee2mqtt since I added multiline pattern in. Zwave2mqtt logs are still there, but they are also duplicated, same as Home Assistant Core logs.

And yes, I can confirm that the patter above doesn’t work for all containers. Mosquitto logs are getting over aggregated, since they don’t start with the timestamp.

bertbaron · June 22, 2022, 4:39am

I guess it’s just too tricky to get the default multiline adapter to work. Instead of trying I better spend time on a proper adapter that works well for HA and comes preconfigured for at least the core containers. I hope you can be patient, it might take a few weeks or a little more but I promise it will come.

ivolodin · June 23, 2022, 12:48pm

Sounds good. I’ve waited for years to be able to get HA logs out, so waiting a few more weeks isn’t really an issue.
One thing to note, even once I removed all of the multiline patterns from the config, I’m still getting duplicated messages for all containers. And I think it started after the latest update to the addon. Is it possible that something changed in the addon itself that duplicates messages?

bertbaron · June 23, 2022, 1:23pm

I can’t think of anything. Do you also get duplicate lines without using multiline on your route? Maybe it’s a bug in that adaper.
Other things you might want to look at:

Does the log from logspout show anything weird
Check the configuration as yaml (the GUI editor fooled me more than once)
Set the Debug variable:
```
- name: DEBUG
  value: "true"
```
for me it doesn’t show much extra but it does show 16 pump.getLogs() messages and I have 16 containers running

If nothing helps I would like to know if exactly each message is duplicated or if it seems to be more random.

CentralCommand · June 23, 2022, 3:11pm

Yea they’re each different unfortunately. Here’s what I use for Z2M and nodered in my promtail config:

- match:
    selector: '{container_name="addon_45df7312_zigbee2mqtt"}'
    stages:
      - multiline:
          firstline: '^(\x{001b}|\(node:\d+\)|\[[-_.:a-z0-9]\]) '
- match:
    selector: '{container_name="addon_a0d7b954_nodered"}'
    stages:
      - multiline:
          firstline: '^(\d{1,2} [A-Za-z]{3,4} \d\d(?::\d\d){2} -|\[[-_.a-z0-9]+\]) '

Also just an FYI, Z2M actually logs to a file in its config folder by default (probably either /config/zigbee2mqtt or /share/zigbee2mqtt depending on when you set it up). Once you have an easy way to review the docker logs for addons you should disable that so it stops chewing up extra disk cycles. You can do this by changing “Log Output” to console and blanking out the “Log File” and “Log Directory” fields in advanced settings.

I don’t use Zwave at all so I don’t know that one. That’s pretty weird though, why would they be duplicated?

I actually didn’t find any need to do multiline for mosquitto. The only thing I have for that one is this:

- match:
    selector: '{container_name="addon_core_mosquitto"}'
    stages:
      - drop:
          expression: ".*Saving in-memory database.*"
          drop_counter_reason: Cron

It pops that line in the log all the time so I just drop it. I don’t have to review the logs of this addon much but its not really helpful to have pages and pages of that message in grafana when I do.

EDIT: I just realized looking at my post that Z2M also uses the terminal color code character. So it looks like that pattern mostly works for that addon as well. If memory serves its a bit more then that there because of exceptions. I believe uncaught exceptions just start with node: and then dump a stack trace. Something like that.

pathia · November 14, 2022, 2:11pm

I also want to suppress the addon_core_mosquitto messages. Can you give me a hint where to put this match code you shared ?

bertbaron · November 14, 2022, 3:15pm

The match code form @CentralCommand refers to the configuration of promptail. Logspout does not have such a configuration. I still have it on my todo list to add something like that in a simplified form but didn’t find the time yet.

Logspout can filter logs based on container labels (GitHub - gliderlabs/logspout: Log routing for Docker container logs). Unfortunately with HA we don’t have much control over those, but fortunately home assistant does assign some labels by itself, in particular the label io.has.name might be useful. In your case you might try:

env:
  - name: EXCLUDE_LABELS
    value: io.hass.name:Mosquitto broker

in your configuration. I didn’t fully test this myself yet, but please let us know if it works.

pathia · November 14, 2022, 3:33pm

Thank you, Bert.
Your suggestion works perfectly!

pathia · November 14, 2022, 10:49pm

Can you also think of a way to exclude log entries from hassio_dns ?

bertbaron · November 15, 2022, 9:23am

If you have access to the cli somehow you can try to run something like this to find the labels:

# docker inspect hassio_dns | jq " .[0].Config.Labels"
{
  "io.hass.arch": "amd64",
  "io.hass.base.arch": "amd64",
  "io.hass.base.image": "alpine:3.14",
  "io.hass.base.name": "alpine",
  "io.hass.base.version": "2022.02.0",
  "io.hass.type": "dns",
  "io.hass.version": "2022.04.1",
  "org.opencontainers.image.authors": "The Home Assistant Authors",
  "org.opencontainers.image.created": "2022-04-26 14:37:01+00:00",
  "org.opencontainers.image.description": "Home Assistant Supervisor plugin for DNS",
  "org.opencontainers.image.documentation": "https://www.home-assistant.io/docs/",
  "org.opencontainers.image.licenses": "Apache License 2.0",
  "org.opencontainers.image.source": "https://github.com/home-assistant/plugin-dns",
  "org.opencontainers.image.title": "Home Assistant DNS Plugin",
  "org.opencontainers.image.url": "https://www.home-assistant.io/",
  "org.opencontainers.image.version": "2022.04.1",
  "supervisor_managed": ""
}

The most useful label seems to be io.hass.type. So your configuration should look like:

env:
  - name: EXCLUDE_LABELS
    value: io.hass.name:Mosquitto broker;io.hass.type:dns

I hope this works.

pathia · November 15, 2022, 9:32am

Awesome!. I’m gonna try it right away.

pathia · November 15, 2022, 4:12pm

It works, Bert. Thank you very much. Both messages were really cluttering my logfile.

phaeton · September 4, 2023, 9:00pm

Hoi Bert, since my hass server randomly crashes, I tried to find a way to send logs outside home assistant. I stumbled on your add after figuring out the original Loki addon doesn’t work and unfortunately isn’t maintained anymore.

I first tried to send logs to my grafana account. This didn’t work due to me not understanding how to configure a user name and password env variable. I could find it in the documentation or comments on GitHub.

I now use papertrail. Works fine! But I’d like to use Loki/grafana since I then can use my existing account. Could you maybe help out? How can I configure username and password?

bertbaron · September 5, 2023, 5:58am

Hi Harry,

From what I see Loki by itself does not use any authentication. I guess the instance you are using adds this by a reverse proxy or something. Do you know the authentication method? Is it Basic Authentication or some other method?

Logspout does not have an option to add authentication headers as far as I know but I can have a look into the code this weekend to see if I can add some support.

phaeton · September 6, 2023, 3:34pm

Hi Bert,

the instance I am using is the free environment grafana offers on their website. Looking at the settings I see it is named basic auth with a user and a password. Where the user has 6 digits and the password is a token like this:

glc_eyJvIjoiOTM4NDI3IiwibiI6InN0YIOrLTczMzc1Ny1obC1yZWFkLWhhc3MiLCJrIjoiVEVCY2tzNkg4Mlo0Nm05UjJWuNQ2OXE5IiwibSI6eyJYIJoicHJvZC1ldS13ZXN0LTMifX0=

phaeton · September 6, 2023, 4:24pm

While grabbing the logs I try to filter out some addons. I do it like this in the env configuration:

- name: DEBUG
  value: "false"
- name: EXCLUDE_LABELS
  value: >-
    io.hass.name:Mosquitto broker;io.hass.name:Logspout
    addon;io.hass.name:Z-Wave JS UI

this results in a lot of messages in the log like this:

2023/09/06 18:22:36 pump.Run() event: dea543979b27 exec_die
2023/09/06 18:22:36 pump.Run() event: f9b2230d60d4 exec_create: /bin/sh -c curl --fail http://127.0.0.1:8099/health/zwave || exit 1
2023/09/06 18:22:36 pump.Run() event: f9b2230d60d4 exec_start: /bin/sh -c curl --fail http://127.0.0.1:8099/health/zwave || exit 1
2023/09/06 18:22:36 pump.Run() event: f9b2230d60d4 exec_die
2023/09/06 18:22:40 pump.Run() event: 1fc73a31dae1 exec_create: /bin/sh -c [[ "curl --fail http://127.0.0.1:9541" ] || [ "curl --fail --insecure https://127.0.0.1:9541" ]] || exit 1
2023/09/06 18:22:40 pump.Run() event: 1fc73a31dae1 exec_die

it seems to work. But now I have to also exclude logs from your add-on
Not sure if this is a real error, but i’d thought I’d mention.

Thanks for the help so far.

bertbaron · September 8, 2023, 1:12pm

I just had a look into the code and this is debug logging. Looking into the code I figured that logspout interprets any non-empty value for the DEBUG variable as true:

	if os.Getenv("DEBUG") != "" {
		log.Println(v...)
	}

So I guess these messages will go away when you set the DEBUG variable explicitly to an empty string ("") (not tested) or leave it out.

Hope this helps

bertbaron · September 11, 2023, 2:35pm

Hi Harry, just an update: I had a look at authentication support and it looks quite good. I managed to get it working with a free account on grafana.com. I need to shape the implementation and add documentation but it should be available soon.

phaeton · September 11, 2023, 7:39pm

That sounds promising Bert! Thanks a million

bertbaron · September 16, 2023, 6:27am

Yesterday I ŕeleased a new version of the addon with support for authenticated calls to Loki. Please let me know if this works for you.