How to get to your log after restart/restore

In a thread recently I was reminded that most HA users operate under the assumption that their log is gone as soon as HA is restarted and if they forget to back it up before fixing whatever problem they encountered they can never get back to it. This actually isn’t true. The file /config/home-assistant.log is wiped clean every restart but that’s not actually your log, that’s just a text file that Home Assistant replicates your log to.

In a supervised or HAOS deployment your real log is the system journal. It is not wiped on restart of HA, reboot of the host or even a restore from backup. And you can access it, even in an HAOS setup, here’s how.

Getting to the journal

HAOS - Recommended way

On HAOS the recommended way to get access to the journal is via the SSH & Web Terminal addon. Note that this is the one in the community addons repository, not the SSH addon in the default repository. The one in the default repository does not map the system journal into the addon’s container. You can add it to your HA with the button below:

Open your Home Assistant instance and show the dashboard of a Supervisor add-on.

Then when ssh’ed into this addon you can find the journal of HAOS system in /var/log/journal:

Now there is one catch with this approach. The only way to read the journal is using journalctl. And journalctl isn’t available in this addon because it is alpine based and journalctl isn’t available in alpine based systems. So what you’ll have to do is copy the /var/log/journal folder and everything in it out of here to another linux system which has journalctl available (debian, ubuntu, etc.) You can do this with scp using scp -r root@<HA URL>:/var/log/journal . or by using the SFTP option in the addon.
Note: you will have to set ssh.username to root in the config at least temporarily for either scp or the sftp option to work. You can change it back to non-root after as the addon’s documentation recommends.

On the system you copied it to you can read from this journal by doing journalctl -D <directory of the copied journal>. There’s more on using journalctl below but just remember to add -D <dir> to the commands since the journal you want to read from will be in a non-standard location.

Supervised Deployments

Simple. SSH into the host and run journalctl. The only supported deployments of supervisor are debian based when using journal logging so you should have everything you need already.

HAOS - Alternative (host access)

If you follow this guide you will be able to SSH into the host system directly. From there you can simply run journalctl as normal like on a normal non-alpine linux system.

Word of caution however, do not mess around in here. Changing things or even installing additional packages can mess up your system in unpredictable ways. And as the note at the top of the guide I linked suggests, you will not get support if you mess up your system while doing this.

This is why the other approach is recommended since it can be accomplished entirely from the front-end plus terminal using supported HA access options.

Exploring the journal

If you aren’t familiar with journalctl I recommend starting with the man page. Linuxhandbook also has a pretty good guide.

That being said, one of the things you’ll quickly notice is that there’s a lot in here. All logs from every container as well as the system itself end up in your journal. So that is every addon, supervisor, the supporting plugins, and everything from the OS itself. Which is a lot.

Fortunately, all log entries from homeassistant, supervisor or addons have a field called CONTAINER_NAME set to the name of that container. That’s the docker container’s name but its roughly what you would expect (homeasistant for home assistant, hassio_supervisor for supervisor, addon_<addon slug> for addons, etc.). We can filter on this like so:

journalctl "CONTAINER_NAME=homeassistant"

That will show you only logs from home assistant. You can also see all the logs for a particular addon over all of its restarts/updates like so:

journalctl "CONTAINER_NAME=addon_a0d7b954_ssh"

Just replace a0d7b954_ssh with the slug of the addon you want to look at. You can get the slug by going to your list of addons, clicking on the addon you’re interested in and then copying and pasting from the url at the top. The format of the URL for an addon looks like this:

<HA URL>/hassio/addon/<ADDON SLUG>/info

Alternatively just let journalctl tell you all the values for CONTAINER_NAME and find the one that looks right in the list using this command:

journalctl -F CONTAINER_NAME

Promtail, Loki and Grafana

This guide helps you to explore your logs without any additional software. It also is intended to help people that are in reaction mode trying desperately to figure out what happened while everything is breaking or already broken.

However if you are not in reaction mode and willing to install additional software to make your life easier around logging, I may humbly suggest taking a look at my addons for promtail and loki. These can be used collectively with the grafana addon in the community add-on store to form the PLG stack. Essentially what that means is you’ll be able to see and explore your full system logs from Grafana including complex queries, graphing, alerts, etc., all the things grafana does well. Take a look at the addon documentation for more info on setting it up if you’re interested.

15 Likes

Useful post!

I did this (BTW “these files” is not specific), but journalctl from two different raspberry pi systems (similar h/w) gave:

There must be a better way!
Can journalctl be installed properly, in some way?
Or how to solve this, if that is really not a possibility?

I hardly check my log from HA since most of the time I will see the logs from Portainer.

I guess I figured this line right below it covered that:

But yea what I meant was everything in /var/log/journal

The alternative is host access, it’s covered above. If you set that up then you can just use journalctl. That’s actually one of the main reasons host access via ssh exists, it’s listed as the example use case in the doc.

If you don’t like that idea and really want to stick in an addon then there would have to be a new ssh addon which uses the debian base image instead of the alpine one. I’m not aware of one but it could be built.

Is it possible to access an old log in case of HA in Docker?

Sure, just depends on how you set up logging in docker. Docker has lots of options for logging so where to go depends on how you set your system up. Supervised and HAOS require journal logging but that’s not a general docker requirement.

I am not a big expert in Docker.
Docker was installed by using default procedures on a Debian host.
Step by step I started using this command to start HA in Docker:

sudo docker run -d --name="HA" -v /home/ildar/HA:/config -v /etc/localtime:/etc/localtime:ro  -v /home/ildar/.ssh:/root/.ssh -v /var/run/docker.sock:/var/run/docker.sock --restart unless-stopped --net=host homeassistant/home-assistant:stable

“ssh” part was added to support access to other *ix hosts in LAN via command_line sensor from HA.
“docker.sock” part was added to support some “docker sensor” (gathering info about containers) - this functionality was not implemented by me in HA, but this part was added to the command.

If you need any additional info - I am ready to provide it.

Thanks for the addons. I tried starting loki with no additional configuration and I get:

mkdir /data/loki/chunks: permission denied

Strangely enough, I tried doing it on my unraid server first and I also got the same message. Any ideas? HA OS on Odroid N2+

edit: congrats on the baby and I see you have a workaround here Permission error creating chunks directory · Issue #167 · mdegat01/addon-loki · GitHub

didn’t work for me

Excactly what I was looking for (because of Sudden HA Core restart after 72 days of uptime - looking for the reason).

Unfortunately after copying the .journal to another system (running Raspberry Pi OS), I only get

journalctl --file="[email protected]"
Failed to open files: The protocol is not supported

End of the road. No logs for me… or is there any smart way of using the journalctl of the HA OS server itself? Would shorten up the whole process a lot… I mean, I can sudo apt install nano in a container, why not journalctl (which will be removed anyway once the container gets restarted/recreated)?

Yes, that’s this section. One of the main reasons a host ssh option exists on HAOS is to get to the system journal.

That being said, I need to update this guide. Recent releases of supervisor have added a much easier option. Now you can simply do something like this:

ha host logs --identifier homeassistant

the host logs command returns results from the system journal by talking to the host using the systemd-journal-gatewayd service. It doesn’t have as many options as journalctl so you may still prefer to find a way to use journalctl that works for you of those at the top. But It does return results for a particular container across all the restarts of that container, including homeassistant.

You can also specify how many lines you want to see with --lines or a particular reboot of the host by something like this;

ha host logs --identifier homeasisstant --boot -1

This returns logs from the homeassistant container from the previous boot of the machine.

I mean you can in theory. The problem is that journalctl is part of systemd. And systemd only exists for debian, not alpine. Most of the addons for HA (including all the SSH ones) are alpine-based. So you cannot install systemd or journalctl, its not available.

You could build an addon with that available though. You could make one using a debian base image and then install the packages needed to get and use journalctl. And then ask supervisor to map the system journal into the container via the addon option.

1 Like