HA not reachable and running out of space

Configuration
1

Hello everybody,
it occurred twice: my HA was not reachable, when I was far from home.
So my first worry was to bring it back to work rather than investigating root causes.

But I can’t ignore the problem, which has no evident clues to me.
So let me share all the infos I have, asking for your help

  1. HA is running on Proxmox: I know it is not recommend and supported, but this allows me to separate phisical resources between HA, Frigate (which runs separatelly) and Jellyfin. The underlying server is connected to the router with a cable, external access happens via Cloudflare and I had not network issue.

  2. In normal conditions HD has plenty of space (50% free space)

    image
    image731×206 10.5 KB

HA uses less than 7% of 2 CPUs and more or less 3 out 6Gb of RAM

image
image611×244 7.83 KB

  1. When the problem came out I was unable to reach with any device (Android smartphone, iPad or PC)

  2. The only thing I was able to do was accessing Proxmox and reboot HA VM

  3. After VM reboot HA seems to be running out of space: disk metrics is almost full with “Home Assistant” (light blue bar) using all of the available space.

  4. In a rush to restore normal condition I have rebooted HA and it came back to normal disk space free condition.

image
image612×270 12.2 KB

So my questions are:

A. What would you suggest to investigate to find the root cause of the issue? I just have an “emergency procedure”, but want my system to be more reliable!
B. Why does the disk space run out and, why is the second reboot (the one from within HA) to clear up everything?

I know I did not investigate properly in the heat of the moment, but trust me: you want to know your home is “up and running”, when you are away! But any evidence is gone, now.

If you find this case interesting, please help me to solve the problem!
Thanks in advance!

2

Running HAOS as a virtual machine is supported.

Start by looking through the logs for clues. Also a 30GB disk is a bit too little, the minimum recommended is 32GB. If you have some spare disk try expanding it.

3

Having any logs set to debug might cause the logs to swell fast.

Also backups are hefty storage users.
The back is created on the storage, even if it is later moved to an external site.
If it fails, then the backup file created on the internal temp backup place might be left behind and take up space.

4

For the sake of completeness, the VM is compliant to the minimum HW requisites. I did not assign more HDD space because in normal conditions I have more than 18Gb free.

image
image1684×411 55.8 KB

I went through Core and Supervisor Logs but found nothing usefull, after HA restart: could this restart have deleted logs?
Thanks

5

Mmmmh this sounds interesting: in fact I had

logger:
  default: debug
  logs:
    custom_components.husqvarna_automower_ble: debug
    custom_components.husqvarna_automower_ble.config_flow: debug
    automower_ble.mower: debug

But I have always had this configuration!
Anyhow I have switched to:

logger:
  default: warning
  # default: debug
  logs:
    custom_components.husqvarna_automower_ble: debug
    custom_components.husqvarna_automower_ble.config_flow: debug
    automower_ble.mower: debug

And the husqvarna_automover is disabled at the moment!
Thanks

6

Restart rotates the logs, log.1 is the file from pre-restart.

7

Yes, homeassistant.log.1 will be deleted.
Homeassistant.log will be renamed to honeassistant.log.1.
A new empty home assistant.log will be created.

8

Hello GitWally,

If you have about 15gb of HA stuff and files, the backup is generated on the tmp folder in that same drive and could well be that same size (or bigger). Then if memory demand is suddenly high, the swap file grows as well eating more space.

I would recommend twice as much free space as space used for safe operation, so if you have used 1/3 of the drive space, add drive space.

9

In Linux swap is a fixed size partition, which does not affect the normal data drive.

But are correct in your recommendation of space, since the backup is created in the tmp folder and then copied to the final storage place and if that is local, then it will at during this copying take up twice the space of the backup.

10

I’m not sure, but I thought the pi uses a file, but probably fixed size, yes.

11

I think you are right.
My HAOS list a swap file as being used with the cat /proc/swaps and also with swapon -s, so it probably is a swap file and not a swap partition.