Any advice how to troubleshoot this would be welcomed.
My HA (OS8.1, core 2022.5.5) which is freshly built (I have HACS installed with EdgeOS, BLE integrations and 10 smart plugs via Conbee) but OS keeps crashing consistently after 1-2 days of rebooting. Disk usage is less than 5GB out 32GB. Same thing happened with OS 7.6.
I do not see anyone else complaining about it so makes me wonder if this is somehow related to Synology 7.0.1 and VM I’m running it?
Errors that keep repeating in 1000’s are (with varying numerical values):
sd 0:0:0:0: [sda] tag#0 access beyond end of device
systemd-journald[108]: Failed to write entry (12 items, 345 bytes), ignoring : Read-only file system
I have also seen on console the following message: ata1.00: failed command: FLUSH CACHE
Any ideas what to check and why this keeps happening.
Is there a setting I need to change in Synology VM that will stop this?
I have not been able to get ssh access to actual OS that runs the HA Docker to see what’s going on.
Does anyone know how to ssh into the OS running HA Docker, as it seem like there is something going with systemd-journald. I install SSH within HA but it did not give me proper access as docker didn’t exist in system I accessed via ssh and there were no logs at all in /var/log/
I only have HDD’s on the Synology and no m.2 cache drives etc.
Host is running number of other Docker images so never sleeping etc.
Disk utilization is around 50% on weekly averages. Using Synology Hybrid Raid (SHR) with a single volume. I do have 10GB network card installed but it is not being used by VM.
Ok looks like system crashes about every 12h.
And I can see these messages in VM Console (on browser)
These are just scrolling continuously on console and the only way to recover is to force shutdown from VM manager.
If anyone knows how to ssh to the actual HA (Hassion) OS where Docker is and journald processes are running I might have a change to try to figure out why it keeps crashing.
I guess one option would be try to implement system reboot every 12h until I figure out permanent fix.
This is very frustrating as I really like HA, really shame that Container version doesn’t allow Supervisor and custom stuff as I didn’t have any issues when running it on Docker while testing HA if it is right solution for me…
Also do not really want to run it on its own hardware, but I might have to just to see if is more stable compared to Synology VM.
I just do not understand why no-one else seems to have this issue, so what’s it with my system or the setup I have.
Do you have HDD sleep enabled? Even if it’s not able to do it due to all Dockers/VM’s.
The only place where those errors popped was HDD spindown and on Proxmox running Xpenology…
I had hibernation time set to 30 mins, which should never happen as system is running several Docker images which are continuously pulling data from my network devices.
Anyway I have disabled it now, I really hope that will fix the issue.
Thanks for your help, much appreciated.
I just noticed there is new DSM 7.1 Update 2 firmware also available, maybe I will do that well in a few days.
What is the model of your Synology box?
Or did you mean you run Synology 7.0.1 as a VM, under… Proxmox?? Those errors looks like disk I/O related.
If you are talking about the VMM that is under Synology OS, the half-measured VM implementation of Synology was one of the reasons I stayed away from their boxes.
That said, if the HA Container works just fine, maybe that’s the option for you. Supervisor is nice and easy for beginners, but not absolutely required. You could absolutely roll and manage those add-ons yourself as your own Container environment.
I have DS1821+ (DSM 7.0.1) and running VM’s from Virtual Machine Manager (and yes I agree with you that it is very flaky, especially with USB devices which are frequently dropping… obviously assuming it is related to VMM, but cannot be for 100% sure).
Ok interesting I did not know you could add custom integrations without Supervisor.
I have several Bluetooth sensors for temp/humidity and I couldn’t figure out how to connect them without BLE integrations via HACS and also EdgeOS to pull data from my network devices.
Yes I’m very new to HA and I was wondering if would you happen to have any links how to get started with adding custom add-ons using Container environment?
(A)
I recall there is a big thread on how you install HAOS under Synology VMM, so it is definitely possible. Maybe you could get some help on your disk-io issue over that thread, or read along the posts and adjust your HAOS VM config/method accordingly (you know, maybe you are close and just one check box away).
Found it:
The HAOS VM, in the background, is actually also appliance OS plus a bunch of hidden dockers. So the HA core is a docker, the supervisor is also a docker, and there are a bunch of service dockers, all packaged nicely into the appliance OS. Furthermore, those add-ons you would get using the HAOS install, those are also dockers.
So this is not for everyone. But then you seem to know your way around dockers. And if you want maximum controls over your thing, sure you could roll those dockers yourselves and tie them together.
I recall there is a user who maintain dozens of docker install scripts on various services - an impressive list covering a lot of grounds around your smart home / home services.
Unfortunately I can’t remember his name just yet. Maybe other users passing by would have better memory (or forum search skills) than I do.
(C)
Again per the HA “Getting Started” install page, there is yet another “Supervised” install method, which would get you the Supervisor, but there will be even more hoops you would have to jump through. Essentially you need to roll a virgin debian VM inside your VMM, and then install HA in the “Supervised” way inside that debian VM, and then meet those requirements.
Again, not for everyone.
Last night I also upgraded DSM to “7.1 Update 2” and VMM to 2.6.0-12122.
So far system has not showed any error messages (like that flush cache error, which was showing every few hours) and everything seems to be better.
I wonder if Synology uptime could have anything to do with VMM stability (it had up time around 5 months before upgrade)? I was able to run HA without any of those issues for about month and then it was constant crashes (less than 12h before HA totally crashed) even with fresh HA install?
I have now configured two VM’s of HA which are running simultaneously, one running with SATA and other using IDE disk settings to see if there is any difference.
Obviously cannot configure both VM’s to use USB devices so one of them is pulling all the network data (external router and couple access points with over 40 devices) and the other VM is connected to BLE/Zigbee sensors, these should have similar impact on file system but might take “twice as long” compared to a single HA.
I suspect I will have to keep monitoring the system for while before convinced it might be working ok.
I will update things here about the progress.
Thanks for everyone providing useful feedback and helping me to learn more about HA, much appreciated!
OK. lt has been almost 3 months since last post on this topic and I think the problem in my case is somehow related to Synology VMM and the DSM 7.
I have noticed that after about 50 days I have to reboot the NAS as I started to get similar issues as previously, but reboot seems to fix them.
I have no idea what is causing them but for now if it happens I will just reboot the NAS. Hopefully future DSM and VMM versions make it more stable.
I’m getting the same problem on a QNAP NAS running Supervised HA under Virtualization Station. This install had been stable for well over a year and has started exhibiting this behavior in the last month or so. The VM has a 512 GB disk. The disk is not dynamic and has no caching enabled.
Has anybody found the cause and/or a solution to this?
