Additional info:
Home Assistant 2022.10.3
Supervisor 2022.10.0
Operating System 9.2
Frontend 20221010.0 - latest
Oke, thank you very much for your reply. I’ll do a search for the CLI commands for syslog files and their location(s).
You can’t with the OS the method that you’re using. Sorry, I wasn’t clear. You can only change the log location if you run the Home Assistant Core installation method. That does not include Home Assistant OS, Supervised, or Container (well you could get it working in container too).
1 Like
I made this addon a while back:
I was using it to ship logs to Loki but it could ship logs to wherever you want. Just aim it the url of a service which supports it.
Or if there’s an alternative log shipping utility you like better you could probably make your own addon version of it in a similar way.
2 Likes
Thank you again for your elaboration in regards to my specific Home Assistant installation setup. So I hope some developer(s) could perhaps introduce an integration for setting up the log-file to remote sys-log in the Home Assistant GUI and so this may well be a feature request as well. Have a nice day.
1 Like
Thank you for prompting me to your Add-on. Appreciated.
1 Like
Agreed, this is a standard option on most autonomous network servers. Not really hard to implement either on sender or receiver end. Few lines of python code.
I’m not sure about solution described below, however I have found that recent (last year or so) updates to Home Assistant log file handling code made it difficult if not impossible to capture the last log entries at crash or HA shutdown/restart. These are often critical to debugging issues. I think any wrapper around a text file monitoring solution/hack has the potential to not capture all log records. The core system has to make sure it writes these last will and testament messages before dying or exiting. The roll over of one prior log file helped a little, however the login history of unix/linux logging shows that more control over logging options should be standard here.
IMHO, log messages in Home Assistant could use some love in general. They often do not point to the source of the message, just the error. And dumping raw python multiline exceptions into the log file make overall monitoring of server painful.
1 Like
Are you referring to the addon I linked with this? If so then you should probably take a look at it. The promtail addon I made does not look at that text file at all. IMO that text file is just unnecessary disk I/O, I really wish I could entirely disable it.
The promtail addon scrapes and ships the system journal of the system it is running on. So it ships not only all HA logs without fail it also has all logs from all containers (supervisor, addons, and the plugins) as well as the logs from the system itself (OS Agent, kernel, systemd, etc.) It gets everything across all reboots of the host and restarts of the various applications, startups and shutdowns included.
1 Like
Hi Mike @CentralCommand I will explore your tool more, I did read through the docs a little. I am skeptical that any external solution will catch all log records. I have spent many years head scratching on the lack of log data from systems/servers. Any ‘tail’ wrapper is going to be dependent on a combo of the source sending and flushing messages before it dies and the host operating systems file i/o close and flush methodology. This is why remote syslog options is very useful, the simple of a send of a udp message as LWT yields the highest level of success I have found (maybe second to a high speed video capture of the console log (I kid you not 
network stack and character mode console output are the last two to die most often)) . A question for you, why can your tool only capture dmesg on x86? dmesg seems to be a pretty low level log output, wondering if this is not an example of the problem with any external solution. Look forward to exploring your tool as a patch, however I think one of core reasons for WTH is surface work arounds and patches that should be core to HA. And again remote syslog is a very simple improvement IMHO.
1 Like
It’s not actually that though just to be clear. On any HA system with supervisor it is a requirement to set up journal logging on the host and to set docker to use it as well. So all HA systems with supervisor are logging everything there.
The promtail addon compiles the promtail binary with journal reading capabilities and the journal is mounted into the container so it can see everything. When you first start it up it immediately starts shipping all the existing logs to wherever you pointed it. And if the addon is restarted then it picks up where it left off by shipping all logs that occurred since then.
Now you can miss logs if the remote service is down temporarily. But thats a hazard in any remote syslog situation.
I’m a bit confused by this question since dmesg isn’t involved here. Are you asking why i386 isn’t supported by the addon? Mainly because loki doesn’t publish a binary for that here and I didn’t have that type of system available to experiment and figure out how to get one working myself.
It’s not as easy as you think. HA core could do it sure. I mean essentially they already do that since logs are both output to docker and replicated to that text file. And if that’s all you want then that’s fine.
However if you want all the logs, not just core but also addons, supervisor, plugins and the host (everything that appears in Settings → System → Logs) then that’s trickier. To replicate that solution every component would have to implement support remote syslog. And ideally in a way so you can tell supervisor what to do and it configures the options needed on everything.
Alternatively there might be a way to make some changes on the host itself to get this to happen. Either config changes to existing software or new software installed that supervisor can change over DBus upon user request. Not just a simple python change though, requires some research.
1 Like
To get syslog (and syslog forwarding) I’ve been using an Home Assistant Supervised instead of the full HAOS install.
Syslog forwarding and hardware monitoring are the main reasons I’ve stuck to using Home Assistant Supervised.
I’ve figured if anyone comes up with an Add-on for rsyslog or similar, I might switch to the full HAOS install. It’s been maybe two years and I haven’t seen such an add-on, so maybe it isn’t possible to do that with the way the base HAOS is configured.
I will admit, I’m still using syslog and a centralized log host based on a habit that is now a couple of decades old. I realize I should look to see what modern approaches (ELK stack, etc.) would be better. I hadn’t heard of Loki before.
1 Like
@CentralCommand I think we are talking a bit in parallel universes here, apologies. You always need to take a step back and understand which of the four installation methods a person is using with Home Assistant, as the experience and options are different. I have only every run HA in a Docker container, first on a Raspberry Pi and now on a Ubuntu x64 machine. I have no experience on the other install methods. On the Docker container model where I am, there is a single log file, home-assistant.log , everything dumps into this file. I don’t think I have ever seen any more information in system menu view of logs that you reference. And there are no other log files that I know of for Home Assistant in my install method.
@rct interesting, I did not know that any of the install methods provided syslog forwarding, can you point me to the document to set this up in your install method? thx!
1 Like
Oh. Fair enough. Yea the OP mentioned running home assistant supervised so I was going off that. There are no add-ons in a docker install so my add-on won’t work there.
Well that’s not entirely true. The Home Assistant container sends logs to stderr of the container. This is typical for most dockerized applications as it allows the admin to decide how to handle logs by choosing a logging driver. So you also don’t really need to look at that text file, just see where docker sends logs currently and potentially redirect it.
In fact when researching how to make the promtail add-on the first thing I noticed is that Promtail recommends against doing what I did in a docker setup (running the binary to scrape and ship the logs). Instead they recommend using their logging driver. Unfortunately this is not an option for any HAOS or supervised user because the only supported configuration is journal logging (also I tried and everything blew up spectacularly 
)
But in a docker install that’s definitely what I would do. Or use a different logging driver if you prefer, I’m sure there’s lots of them out there for whatever tech stack you use.
2 Likes
Sorry for any confusion, I didn’t mean to imply that syslog forwarding was included.
With Home Assistant Supervised you control the OS, so you can do whatever you want with syslog or the rest of the OS.
With the “supervised install”, you are running the Home Assistant Supervisor in a container. The supervisor then manages the Home Assistant container(s), and any add-on containers.
I have only every run HA in a Docker container,
[…]
On the Docker container model where I am, there is a single log file, home-assistant.log , everything dumps into this file.
[…]
You always need to take a step back and understand which of the four installation methods a person is using with Home Assistant
If you are only running the Home Assistant (core) container, then yes there is only a single log for file Home Assistant Core application. I was referring to running the Home Assistant Operating System. So there’s a kernel running and it’s “dmesg” output is what I’d like to forward to be able to see if/when hardware related log messages pop up.
I’m seeing the Home Assistant log entries forwarded to my syslog server. I was going to say I’m not sure what is doing that, but it looks like @CentralCommand just addressed that. I don’t know if docker-ce came with the logging driver that’s forwarding to syslog out of the box, or whether that’s something installing the HA supervisor did. But I’m seeing the STDERR from all containers running on that host forwarded to my syslog server.
3 Likes
Yea I mean it basically has to be this way. If HA was only logging to the text file then doing a docker logs homeassistant would show you nothing.
1 Like
Hey @rct thank you! You have just taught me something and I am going to try and apply to my setup. Thinking about your setup, I google whether I could get my docker system to forward the output of a container log to rsyslog and it appears so. And my Home Assistant seems to push its message to the docker container log the same as to it’s home-assistant.log . Looks simple, I need to restart docker so that will have to wait for a maintenance window. I am interested to see if this give me another way to look at the full and updated history. Again, thanks for opening my eyes! One of those duh moments that make having other sets of eyes so helpful!
2 Likes
You and @rct both opened my eyes to what appears to be good opportunity for docker based Home Assistant, letting Docker sent the containers output to rsyslog, thanks:
4 Likes
Thank you very much @dproffer @CentralCommand @rct for all your inputs, suggestions and comments. Great to read all your insights about behind the scenes of syslog in general and Home Assistant in particular. As previously mentioned I run a SynologY NAS VMM based Home Assistant Supervised, so I was searching for a ready to install add-on.
But reading your posts, things are not as simple as I’ve imagined. Is it possible for me to use the Docker logging driver to pull all logs from within my VMM Home Assistant: Core / Supervisor / Host / InfluxDB / Grafana etc?
@dproffer Thanks for providing the link to the Docker Documentation.
Nope. Like I said, not supported and blows it up.
You’re welcome to try other ones obviously. But definitely not supported to do so and may cause supervisor to crash and not start again. Be prepared to get out of that, can’t just restore a backup in that situation.
1 Like
I had the same use case (wanting to send HA logs to Synology NAS), and I found this: GitHub - TheByteStuff/RemoteSyslog_Service: Home Assistant Custom Component - send Syslog message to remote server.
You need then to set up an automation, like this (you can customise the message etc.):
alias: Syslog to Syno
description: ""
trigger:
- platform: event
event_type: system_log_event
condition: []
action:
- service: remote_syslog.logmessage
data:
loglevel: >
{% set lvl = {'DEBUG':'debug', 'INFO':'info', 'WARNING':'warning',
'ERROR':'err', 'CRITICAL':'crit'} -%} {{ lvl[trigger.event.data.level]
}}
host: [your.syno.IP]
message: >-
{{ trigger.event.data.level ~ " " ~ trigger.event.data.timestamp |
timestamp_local ~ " : " ~ trigger.event.data.message ~ "< " ~
trigger.event.data.exception ~ " >" ~ trigger.event.data.source }}
program: "{{ trigger.event.data.name }}"
port: 514
protocol: UDP
facility: USER
mode: queued
max: 30
In your config file, you must first set this to have events firing:
system_log:
fire_event: true
And then you have to set up the receiving end on your Synology NAS.
3 Likes
TIL you can tell HA to fire an event for every logged message. Which I guess makes this WTH a lot easiser since you can send the details of those events wherever you want via an automation.
How big is your DB though? There’s an events table in the DB and every event is logged there. Seems like having each logged message also create a row in the DB would cause it to grow quite rapidly. Or do you exclude this event type in recorder?
1 Like