Running out of space on Home Assistant Green

ncksol · April 1, 2025, 10:31am

I’m basically to 0% free space on my HA Green machine. I’ve started investigation today and came up empty trying to figure out who is doing it.
Here is what I have done so far (thank you ChatGPT for guidance):

du -sh /
says I have 1.2G used in /
df -h /
says that 100% is used

Filesystem                Size      Used Available Use% Mounted on
overlay                  28.0G     27.5G         0 100% /

My estimated database size is only 235.88 MiB
Next I went to check which docker images I have and result is:

Docker images are taking a chunk of space (~6.8 GB total), mostly:
Home Assistant core image: 1.83 GB
VSCode add-on: 1.06 GB
Grafana: 815 MB
Samba: 108 MB
Supervisor: 434 MB
Docker containers and volumes are taking very little space — under 200 MB total
No Docker build cache (0 B)
No Docker volumes using meaningful space

So only 6.8 GB in docker. Still about 20 GB unaccounted for.

Searched for big log files or deleted-but-open files

find /proc/*/fd -ls 2>/dev/null | grep '(deleted)' | sort -k7 -rh | head -n 10

returned nothing

du -h -d1 /var | sort -h

returned:

4.0K    /var/db
4.0K    /var/empty
4.0K    /var/local
4.0K    /var/mail
4.0K    /var/opt
4.0K    /var/tmp
8.0K    /var/spool
12.0K   /var/cache
28.0K   /var/lib
496.3M  /var
496.3M  /var/log

logs are slightly on the big side but not the amount I was looking for.

Next ChatGPT suggested that the space is eaten by Docker overlay storage. Which I have found is in:
/mnt/data/docker
but trying to examine that folder gives me an error that it doesn’t exist.
Next ChatGPT proceeded to try to get me to get access to that folder through different means, including SSHing to the HA via external terminal. But no success.
I’ve also tried docker system prune --volumes -a with no success. And also tried restarting the host machine.

Anything else I can do to clean up my HA Green machine?

NathanCu · April 1, 2025, 11:36am

Barking up the wrong tree with the deep core there it’s probably way less hidden. Number one and two with a bullet are media and backups. Look n your media folder, how many backups are you storing. Do you have any security cameras doing things in the background. Then database size.

I see you’ve got some heavy hitters like grafana where are you storing the data.

You can also uninstall VS code and connect remotely from your desktop (connect to the /config folder through smb) to save almost two gigs there.

robertklep · April 1, 2025, 11:42am

Can you also post:

du -h
docker system df

ncksol · April 1, 2025, 1:24pm

du -h produces a really long list of files, do you need all of it? the sum in the end is “1.2G”
docker system df gives the same info that roughly around 6GB is used for docker images and other things.

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          13        13        5.539GB   573.9MB (10%)
Containers      13        13        115.8MB   0B (0%)
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B

ncksol · April 1, 2025, 1:26pm

I’ve checked backups, nothing big there. Same with media. Here is the output from du -hs *, none of the folders from the output are big enough to occupy 28GB of space:
du -hs *

8.0K	addon_configs
4.0K	addons
67.6M	backup
3.2M	bin
52.0K	command
0	config
40.0K	data
0	dev
1.7M	etc
12.0K	home
333.7M	homeassistant
4.0K	init
1.6M	lib
4.0K	lib64
1.0M	media
4.0K	mnt
4.0K	opt
10.1M	package
0	proc
14.7M	root
632.0K	run
756.0K	sbin
4.0K	share
4.0K	srv
4.0K	ssl
0	sys
76.0K	tmp
316.6M	usr
489.7M	var

robertklep · April 1, 2025, 1:32pm

Sorry, I mean df -h

ncksol · April 1, 2025, 1:52pm

Here’s the output from df -h

Filesystem                Size      Used Available Use% Mounted on
overlay                  28.0G     27.6G         0 100% /
devtmpfs                  1.6G         0      1.6G   0% /dev
tmpfs                     1.8G         0      1.8G   0% /dev/shm
/dev/mmcblk0p8           28.0G     27.6G         0 100% /media
/dev/mmcblk0p8           28.0G     27.6G         0 100% /backup
/dev/mmcblk0p8           28.0G     27.6G         0 100% /share
/dev/mmcblk0p8           28.0G     27.6G         0 100% /addons
/dev/mmcblk0p8           28.0G     27.6G         0 100% /homeassistant
/dev/mmcblk0p8           28.0G     27.6G         0 100% /data
/dev/mmcblk0p8           28.0G     27.6G         0 100% /addon_configs
/dev/mmcblk0p8           28.0G     27.6G         0 100% /ssl
/dev/mmcblk0p8           28.0G     27.6G         0 100% /etc/asound.conf
tmpfs                     1.8G         0      1.8G   0% /dev/shm
tmpfs                   734.6M      1.8M    732.9M   0% /run/dbus
/dev/mmcblk0p8           28.0G     27.6G         0 100% /run/audio
tmpfs                   734.6M      1.8M    732.9M   0% /run/docker.sock
/dev/mmcblk0p8           28.0G     27.6G         0 100% /etc/resolv.conf
/dev/mmcblk0p8           28.0G     27.6G         0 100% /etc/hostname
/dev/mmcblk0p8           28.0G     27.6G         0 100% /etc/hosts
tmpfs                   734.6M      1.8M    732.9M   0% /run/log/journal
/dev/mmcblk0p8           28.0G     27.6G         0 100% /etc/pulse/client.conf
/dev/mmcblk0p8           28.0G     27.6G         0 100% /var/log/journal
tmpfs                     1.8G         0      1.8G   0% /proc/asound
devtmpfs                  1.6G         0      1.6G   0% /proc/keys
devtmpfs                  1.6G         0      1.6G   0% /proc/timer_list
tmpfs                     1.8G         0      1.8G   0% /proc/scsi
tmpfs                     1.8G         0      1.8G   0% /sys/firmware

robertklep · April 1, 2025, 2:14pm

And mount?

BTW: double post? Home Assistant Green: Running Out of Space

ncksol · April 1, 2025, 2:27pm

Here is the output for mount

overlay on / type overlay (rw,relatime,lowerdir=/mnt/data/docker/overlay2/l/ZBVPVVDRBKICWEPSNA3S4QHWK4:/mnt/data/docker/overlay2/l/IIFJQ3TYXJQWNTZ55753X4CT2B:/mnt/data/docker/overlay2/l/JFSWOVBRJFF7XZHSWNTTHUAY5U:/mnt/data/docker/overlay2/l/K4BDFXTLCHWWVDNYET3AAYTKXC:/mnt/data/docker/overlay2/l/GZMN7UADK7BD75OZNXZOGUQL5F:/mnt/data/docker/overlay2/l/GNHJE7GNYOA2KCVDM6BJEAJSRW:/mnt/data/docker/overlay2/l/7GNIL4TOMERH3MBG3SWJJBXKDZ:/mnt/data/docker/overlay2/l/4IGZLGKMYWUP7HU7F26AWBMALL:/mnt/data/docker/overlay2/l/SLN4Z7NEIFT6JG3HJ3HDBATARW,upperdir=/mnt/data/docker/overlay2/52c775de52c06d470ede718b97d2f89389dfc12cb6600ef2793309a982a5412c/diff,workdir=/mnt/data/docker/overlay2/52c775de52c06d470ede718b97d2f89389dfc12cb6600ef2793309a982a5412c/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
devtmpfs on /dev type devtmpfs (ro,relatime,size=1681744k,nr_inodes=420436,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/mmcblk0p8 on /media type ext4 (rw,relatime,commit=30)
/dev/mmcblk0p8 on /backup type ext4 (rw,relatime,commit=30)
/dev/mmcblk0p8 on /share type ext4 (rw,relatime,commit=30)
/dev/mmcblk0p8 on /addons type ext4 (rw,relatime,commit=30)
/dev/mmcblk0p8 on /homeassistant type ext4 (rw,relatime,commit=30)
/dev/mmcblk0p8 on /data type ext4 (rw,relatime,commit=30)
/dev/mmcblk0p8 on /addon_configs type ext4 (rw,relatime,commit=30)
/dev/mmcblk0p8 on /ssl type ext4 (rw,relatime,commit=30)
/dev/mmcblk0p8 on /etc/asound.conf type ext4 (ro,relatime,commit=30)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/dbus type tmpfs (ro,nosuid,nodev,size=752264k,nr_inodes=819200,mode=755)
/dev/mmcblk0p8 on /run/audio type ext4 (ro,relatime,commit=30)
tmpfs on /run/docker.sock type tmpfs (ro,nosuid,nodev,size=752264k,nr_inodes=819200,mode=755)
/dev/mmcblk0p8 on /etc/resolv.conf type ext4 (rw,relatime,commit=30)
/dev/mmcblk0p8 on /etc/hostname type ext4 (rw,relatime,commit=30)
/dev/mmcblk0p8 on /etc/hosts type ext4 (rw,relatime,commit=30)
tmpfs on /run/log/journal type tmpfs (ro,nosuid,nodev,size=752264k,nr_inodes=819200,mode=755)
/dev/mmcblk0p8 on /etc/pulse/client.conf type ext4 (ro,relatime,commit=30)
/dev/mmcblk0p8 on /var/log/journal type ext4 (ro,relatime,commit=30)
sysfs on /sys/class/gpio type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/asound type tmpfs (ro,relatime)
devtmpfs on /proc/keys type devtmpfs (ro,relatime,size=1681744k,nr_inodes=420436,mode=755)
devtmpfs on /proc/timer_list type devtmpfs (ro,relatime,size=1681744k,nr_inodes=420436,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime)
tmpfs on /sys/firmware type tmpfs (ro,relatime)

And no that other post is not mine, but what a coincidence lol

robertklep · April 1, 2025, 3:00pm

I’m not sure I understand the setup here. I don’t have an HA Green so can’t check, but it seems weird that it uses /mnt/data/docker/... as lower and upper dirs for an overlayfs mounted on /

AFAIK overlayfs “hides” the lower and upper directories so you can’t individually check them without unmounting the overlayfs first, but I don’t know how that would work in an HA Green

ncksol · April 1, 2025, 3:03pm

Thanks for checking this for me. This is concerning. I will continue to dig

ncksol · April 1, 2025, 6:54pm

I ended up removing addons for VSCode, Grafana and InfluxDB and got the space back. So, one of them was polluting somewhere and not cleaning up after itself… We will never know who it was