HA OS Began Crashing on Unraid/qemu - qemu_thread_create: Resource temporarily unavailable

Hello,

I have been running Home Assistant for years and it has been rock solid. As of a few days ago I randomly find my VM in a “stopped” state with the last item in the VM’s logs being “qemu_thread_create: Resource temporarily unavailable”

More full logs from the VM are:

-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"ich9-usb-ehci1","id":"usb","bus":"pcie.0","addr":"0x7.0x7"}' \
-device '{"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pcie.0","multifunction":true,"addr":"0x7"}' \
-device '{"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pcie.0","addr":"0x7.0x1"}' \
-device '{"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pcie.0","addr":"0x7.0x2"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/HomeAssistant/haos_ova-9.5.vmdk","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"vmdk","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.3","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk2","bootindex":1,"write-cache":"on"}' \
-netdev tap,fd=35,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:ad:6a:98","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=33,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,websocket=5700,audiodev=audio1 \
-k en-us \
-device '{"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/005","id":"hostdev0","bus":"usb.0","port":"2"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.4","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
qxl_send_events: spice-server bug: guest stopped, ignoring
qemu: qemu_thread_create: Resource temporarily unavailable
2024-01-09 18:30:52.940+0000: shutting down, reason=crashed

I was running 11.3, but just updated to 11.4. I looked for logs on HomeAssistants end to see if I could provide more info but was not able to find anything. The logs I did find only seem to persist for the latest boot. If there are logs I can provide to help debug let me know. Thank you!

Using SFTP I was able to download the .journal files, but I cannot manage to open them. I have a MacOS computer and created a Ubuntu docker container but get the error: Failed to open files: Protocol not supported

root@69edeef2ed82:/tmp/6716b6cd39cd9a1b0cc465894d539a1c# ls -l
total 511744
-rw-r--r-- 1 501 dialout  8388608 Jan  9 23:54 system.journal
-rw-r--r-- 1 501 dialout 16777216 Jan  9 23:54 [email protected]~
-rw-r--r-- 1 501 dialout  8388608 Jan  9 23:54 [email protected]~
-rw-r--r-- 1 501 dialout 16777216 Jan  9 23:54 [email protected]~
-rw-r--r-- 1 501 dialout 21645152 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000003be1a9-00060aac27597b36.journal
-rw-r--r-- 1 501 dialout 21577328 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000003c3cdf-00060ada205b3977.journal
-rw-r--r-- 1 501 dialout 20450104 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000003ca2fa-00060ae58246a653.journal
-rw-r--r-- 1 501 dialout 21748136 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000003cff91-00060aea0477922b.journal
-rw-r--r-- 1 501 dialout 21651656 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000003d5a92-00060b124c30286e.journal
-rw-r--r-- 1 501 dialout 22467456 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000003db828-00060b3a71398e80.journal
-rw-r--r-- 1 501 dialout 21897848 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000003e1e4f-00060b637c11849d.journal
-rw-r--r-- 1 501 dialout 21821064 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000003e7dc9-00060b90b71b7dd4.journal
-rw-r--r-- 1 501 dialout 23081304 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000003edf6f-00060bb5c7035b7f.journal
-rw-r--r-- 1 501 dialout 21588728 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000003f4e06-00060bdfd645d49c.journal
-rw-r--r-- 1 501 dialout 21317136 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000003fab24-00060c12a24be4d5.journal
-rw-r--r-- 1 501 dialout 21287728 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-0000000000400560-00060c42e56e957b.journal
-rw-r--r-- 1 501 dialout 21182232 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-0000000000405f77-00060c7239aee398.journal
-rw-r--r-- 1 501 dialout 21207240 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-000000000040b839-00060ca3f5215c54.journal
-rw-r--r-- 1 501 dialout 21231752 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-0000000000411208-00060cd20ffac40d.journal
-rw-r--r-- 1 501 dialout 21300280 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-0000000000416b9f-00060d022af424f6.journal
-rw-r--r-- 1 501 dialout 22057464 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-000000000041c634-00060d30fcd7c9dc.journal
-rw-r--r-- 1 501 dialout 21362928 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-000000000042295b-00060d60a0ad0998.journal
-rw-r--r-- 1 501 dialout 21235472 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-00000000004283c9-00060d932401ff69.journal
-rw-r--r-- 1 501 dialout 21172552 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-000000000042dcfb-00060dc4acc995e4.journal
-rw-r--r-- 1 501 dialout 21137952 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-0000000000433594-00060df651a9c3e7.journal
-rw-r--r-- 1 501 dialout 21220192 Jan  9 23:54 system@91a13c1d5b3043a2b66171a19477b249-0000000000438e47-00060e26f5f39d62.journal
root@69edeef2ed82:/tmp/6716b6cd39cd9a1b0cc465894d539a1c# journalctl --file=system.journal
Failed to open files: Protocol not supported