Systemd-journad: Failed to write entry; Synology VM

[Rey]

Hello,

Any advice how to troubleshoot this would be welcomed.

My HA (OS8.1, core 2022.5.5) which is freshly built (I have HACS installed with EdgeOS, BLE integrations and 10 smart plugs via Conbee) but OS keeps crashing consistently after 1-2 days of rebooting. Disk usage is less than 5GB out 32GB. Same thing happened with OS 7.6.

I do not see anyone else complaining about it so makes me wonder if this is somehow related to Synology 7.0.1 and VM I’m running it?

Errors that keep repeating in 1000’s are (with varying numerical values):

sd 0:0:0:0: [sda] tag#0 access beyond end of device
systemd-journald[108]: Failed to write entry (12 items, 345 bytes), ignoring : Read-only file system

I have also seen on console the following message:
ata1.00: failed command: FLUSH CACHE

Any ideas what to check and why this keeps happening.
Is there a setting I need to change in Synology VM that will stop this?

I have not been able to get ssh access to actual OS that runs the HA Docker to see what’s going on.
Does anyone know how to ssh into the OS running HA Docker, as it seem like there is something going with systemd-journald. I install SSH within HA but it did not give me proper access as docker didn’t exist in system I accessed via ssh and there were no logs at all in /var/log/

Much appreciate any help I can get.

[BeardedConti (BeardedTinker)]

Are you using HDD or SSD drives, is your VM volume on main volume or have you used m.2 bay for cache to host VM’s. If not, do you have m.2 cache?

[Rey]

I only have HDD’s on the Synology and no m.2 cache drives etc.
Host is running number of other Docker images so never sleeping etc.
Disk utilization is around 50% on weekly averages. Using Synology Hybrid Raid (SHR) with a single volume. I do have 10GB network card installed but it is not being used by VM.

[Rey]

Ok looks like system crashes about every 12h.
And I can see these messages in VM Console (on browser)
These are just scrolling continuously on console and the only way to recover is to force shutdown from VM manager.

I tried to see if there are any options in VM to try but nothing obvious.

Does failed to flush cache mean anything?

If anyone knows how to ssh to the actual HA (Hassion) OS where Docker is and journald processes are running I might have a change to try to figure out why it keeps crashing.

I guess one option would be try to implement system reboot every 12h until I figure out permanent fix.

This is very frustrating as I really like HA, really shame that Container version doesn’t allow Supervisor and custom stuff as I didn’t have any issues when running it on Docker while testing HA if it is right solution for me…

Also do not really want to run it on its own hardware, but I might have to just to see if is more stable compared to Synology VM.

I just do not understand why no-one else seems to have this issue, so what’s it with my system or the setup I have.

[BeardedConti (BeardedTinker)]

@Rey

failed to flush cache

Do you have HDD sleep enabled? Even if it’s not able to do it due to all Dockers/VM’s.
The only place where those errors popped was HDD spindown and on Proxmox running Xpenology…

Really no idea what else could be the issue here.

[Rey]

I had hibernation time set to 30 mins, which should never happen as system is running several Docker images which are continuously pulling data from my network devices.

Anyway I have disabled it now, I really hope that will fix the issue.
Thanks for your help, much appreciated.

I just noticed there is new DSM 7.1 Update 2 firmware also available, maybe I will do that well in a few days.

[k8gg]

What is the model of your Synology box?
Or did you mean you run Synology 7.0.1 as a VM, under… Proxmox?? Those errors looks like disk I/O related.

If you are talking about the VMM that is under Synology OS, the half-measured VM implementation of Synology was one of the reasons I stayed away from their boxes.

That said, if the HA Container works just fine, maybe that’s the option for you. Supervisor is nice and easy for beginners, but not absolutely required. You could absolutely roll and manage those add-ons yourself as your own Container environment.

[Rey]

I have DS1821+ (DSM 7.0.1) and running VM’s from Virtual Machine Manager (and yes I agree with you that it is very flaky, especially with USB devices which are frequently dropping… obviously assuming it is related to VMM, but cannot be for 100% sure).

Ok interesting I did not know you could add custom integrations without Supervisor.
I have several Bluetooth sensors for temp/humidity and I couldn’t figure out how to connect them without BLE integrations via HACS and also EdgeOS to pull data from my network devices.

Yes I’m very new to HA and I was wondering if would you happen to have any links how to get started with adding custom add-ons using Container environment?

[k8gg]

(A)
I recall there is a big thread on how you install HAOS under Synology VMM, so it is definitely possible. Maybe you could get some help on your disk-io issue over that thread, or read along the posts and adjust your HAOS VM config/method accordingly (you know, maybe you are close and just one check box away).
Found it:

Installation on Synology Virtual Machine Managager

Here are a quick guide to get Home Assistant OS running in Synology VMM. Install Synology VMM (only supported on some models but I managed to get it to run on my DS716+II by downloading directly from Synology dsm6, dsm7) Download the Home Assistan OS VMware ESXi/vSphere image from here: Alternative - Home Assistant. Start up VMM and create a Storage pool. Create a network interface (Choose Type External and select the primary Network Interface). [Screenshot 2021-02-17 at 09.04.04] Click on V…

(B)
Per the HA “Getting Started” install page there are 4 different ways to get HA going. I see you have tried the docker (not supervised) install method, but wanting more (eg custom integrations via HACS or add-on),
However, to install HACS for example does not really require the Supervisor. I found a couple of links that might help:
Download and run the installer inside the container | HACS
How to install & use Home Assistant Community Store (HACS) | TUTORIAL - Kiril Peyanski’s Blog
Can’t install HACS in Homeassistant in Docker on Synology NAS : homeassistant (reddit.com)

The HAOS VM, in the background, is actually also appliance OS plus a bunch of hidden dockers. So the HA core is a docker, the supervisor is also a docker, and there are a bunch of service dockers, all packaged nicely into the appliance OS. Furthermore, those add-ons you would get using the HAOS install, those are also dockers.
So this is not for everyone. But then you seem to know your way around dockers. And if you want maximum controls over your thing, sure you could roll those dockers yourselves and tie them together.

I recall there is a user who maintain dozens of docker install scripts on various services - an impressive list covering a lot of grounds around your smart home / home services.
Unfortunately I can’t remember his name just yet. Maybe other users passing by would have better memory (or forum search skills) than I do.

(C)
Again per the HA “Getting Started” install page, there is yet another “Supervised” install method, which would get you the Supervisor, but there will be even more hoops you would have to jump through. Essentially you need to roll a virgin debian VM inside your VMM, and then install HA in the “Supervised” way inside that debian VM, and then meet those requirements.
Again, not for everyone.

[BeardedConti (BeardedTinker)]

@k8gg

I recall there is a big thread on how you install HAOS under Synology VMM, so it is definitely possible.

That sure is possible. I run 2 instances of HA VM on two different Synology devices

But this issue I’ve never seen on that thread or anywhere else, except Proxmox running Synology/Xpenology. Not saying that I’ve seen all there is.

[BeardedConti (BeardedTinker)]

@Rey

I have DS1821+ (DSM 7.0.1) and running VM’s from Virtual Machine Manager (and yes I agree with you that it is very flaky, especially with USB devices which are frequently dropping… obviously assuming it is related to VMM, but cannot be for 100% sure).

Check once again your settings:

Also, as VMM can’t run on anything else, I guess your files system is btrfs

And check your storage for VM settings.

I use SATA on both devices. Check what you have.

[Rey]

VMM is configured to use IDE Controller.

Last night I also upgraded DSM to “7.1 Update 2” and VMM to 2.6.0-12122.
So far system has not showed any error messages (like that flush cache error, which was showing every few hours) and everything seems to be better.

I wonder if Synology uptime could have anything to do with VMM stability (it had up time around 5 months before upgrade)? I was able to run HA without any of those issues for about month and then it was constant crashes (less than 12h before HA totally crashed) even with fresh HA install?

I have now configured two VM’s of HA which are running simultaneously, one running with SATA and other using IDE disk settings to see if there is any difference.

Obviously cannot configure both VM’s to use USB devices so one of them is pulling all the network data (external router and couple access points with over 40 devices) and the other VM is connected to BLE/Zigbee sensors, these should have similar impact on file system but might take “twice as long” compared to a single HA.

I suspect I will have to keep monitoring the system for while before convinced it might be working ok.
I will update things here about the progress.

Thanks for everyone providing useful feedback and helping me to learn more about HA, much appreciated!