How to get to your log after restart/restore

In a thread recently I was reminded that most HA users operate under the assumption that their log is gone as soon as HA is restarted and if they forget to back it up before fixing whatever problem they encountered they can never get back to it. This actually isn’t true. The file /config/home-assistant.log is wiped clean every restart but that’s not actually your log, that’s just a text file that Home Assistant replicates your log to.

In a supervised or HAOS deployment your real log is the system journal. It is not wiped on restart of HA, reboot of the host or even a restore from backup. And you can access it, even in an HAOS setup, here’s how.

Getting to the journal

HAOS - Recommended way

On HAOS the recommended way to get access to the journal is via the SSH & Web Terminal addon. Note that this is the one in the community addons repository, not the SSH addon in the default repository. The one in the default repository does not map the system journal into the addon’s container. You can add it to your HA with the button below:

Open your Home Assistant instance and show the dashboard of a Supervisor add-on.

Then when ssh’ed into this addon you can find the journal of HAOS system in /var/log/journal:

Now there is one catch with this approach. The only way to read the journal is using journalctl. And journalctl isn’t available in this addon because it is alpine based and journalctl isn’t available in alpine based systems. So what you’ll have to do is copy the /var/log/journal folder and everything in it out of here to another linux system which has journalctl available (debian, ubuntu, etc.) You can do this with scp using scp -r [email protected]<HA URL>:/var/log/journal . or by using the SFTP option in the addon.
Note: you will have to set ssh.username to root in the config at least temporarily for either scp or the sftp option to work. You can change it back to non-root after as the addon’s documentation recommends.

On the system you copied it to you can read from this journal by doing journalctl -D <directory of the copied journal>. There’s more on using journalctl below but just remember to add -D <dir> to the commands since the journal you want to read from will be in a non-standard location.

Supervised Deployments

Simple. SSH into the host and run journalctl. The only supported deployments of supervisor are debian based when using journal logging so you should have everything you need already.

HAOS - Alternative (host access)

If you follow this guide you will be able to SSH into the host system directly. From there you can simply run journalctl as normal like on a normal non-alpine linux system.

Word of caution however, do not mess around in here. Changing things or even installing additional packages can mess up your system in unpredictable ways. And as the note at the top of the guide I linked suggests, you will not get support if you mess up your system while doing this.

This is why the other approach is recommended since it can be accomplished entirely from the front-end plus terminal using supported HA access options.

Exploring the journal

If you aren’t familiar with journalctl I recommend starting with the man page. Linuxhandbook also has a pretty good guide.

That being said, one of the things you’ll quickly notice is that there’s a lot in here. All logs from every container as well as the system itself end up in your journal. So that is every addon, supervisor, the supporting plugins, and everything from the OS itself. Which is a lot.

Fortunately, all log entries from homeassistant, supervisor or addons have a field called CONTAINER_NAME set to the name of that container. That’s the docker container’s name but its roughly what you would expect (homeasistant for home assistant, hassio_supervisor for supervisor, addon_<addon slug> for addons, etc.). We can filter on this like so:

journalctl "CONTAINER_NAME=homeassistant"

That will show you only logs from home assistant. You can also see all the logs for a particular addon over all of its restarts/updates like so:

journalctl "CONTAINER_NAME=addon_a0d7b954_ssh"

Just replace a0d7b954_ssh with the slug of the addon you want to look at. You can get the slug by going to your list of addons, clicking on the addon you’re interested in and then copying and pasting from the url at the top. The format of the URL for an addon looks like this:

<HA URL>/hassio/addon/<ADDON SLUG>/info

Alternatively just let journalctl tell you all the values for CONTAINER_NAME and find the one that looks right in the list using this command:

journalctl -F CONTAINER_NAME

Promtail, Loki and Grafana

This guide helps you to explore your logs without any additional software. It also is intended to help people that are in reaction mode trying desperately to figure out what happened while everything is breaking or already broken.

However if you are not in reaction mode and willing to install additional software to make your life easier around logging, I may humbly suggest taking a look at my addons for promtail and loki. These can be used collectively with the grafana addon in the community add-on store to form the PLG stack. Essentially what that means is you’ll be able to see and explore your full system logs from Grafana including complex queries, graphing, alerts, etc., all the things grafana does well. Take a look at the addon documentation for more info on setting it up if you’re interested.

7 Likes

Useful post!

I did this (BTW “these files” is not specific), but journalctl from two different raspberry pi systems (similar h/w) gave:

There must be a better way!
Can journalctl be installed properly, in some way?
Or how to solve this, if that is really not a possibility?

I hardly check my log from HA since most of the time I will see the logs from Portainer.

I guess I figured this line right below it covered that:

But yea what I meant was everything in /var/log/journal

The alternative is host access, it’s covered above. If you set that up then you can just use journalctl. That’s actually one of the main reasons host access via ssh exists, it’s listed as the example use case in the doc.

If you don’t like that idea and really want to stick in an addon then there would have to be a new ssh addon which uses the debian base image instead of the alpine one. I’m not aware of one but it could be built.