Where did all the storage go or not enough disk space on /data

markuskuttke · October 18, 2020, 5:14pm

Hi,
not a bloody noob on Linux here. So I did a little research also here on the forum regarding my issue but couldn’t find any answers.
So let’s start:
Setting
Running HassOS on a Synology Disk Station with the build in Hypervisor (Virtual Machine Manager). I installed the HassOS by downloading the prepared OVA image/disk. Since I didn’t change anything I left the size of 10GB.
How I got Problems
I run my weekly (manual) update-cycle to see if there were any update for HassOS and Hom Assistant Core. Both had nev versions. So I first upgraded HassOS and then I my intention was to update the Core.
It failed, no big deal:

20-10-18 12:26:17 INFO (MainThread) [supervisor.homeassistant.core] Update Home Assistant to version 0.116.4
20-10-18 12:26:17 INFO (SyncWorker_0) [supervisor.docker.interface] Update image homeassistant/qemux86-64-homeassistant:0.116.2 to homeassistant/qemux86-64-homeassistant:0.116.4
20-10-18 12:26:17 INFO (SyncWorker_0) [supervisor.docker.interface] Pull image homeassistant/qemux86-64-homeassistant tag 0.116.4.
20-10-18 12:31:07 ERROR (SyncWorker_0) [supervisor.docker.interface] Can't install homeassistant/qemux86-64-homeassistant:0.116.4 -> 404 Client Error: Not Found ("no such image: homeassistant/qemux86-64-homeassistant:0.116.4: No such image: homeassistant/qemux86-64-homeassistant:0.116.4").
20-10-18 12:31:07 INFO (SyncWorker_0) [supervisor.docker.interface] This error is often caused by not having enough disk space available. Available space in /data is: 0.5 GiB
20-10-18 12:31:07 WARNING (MainThread) [supervisor.homeassistant.core] Update Home Assistant image failed

So it is low on data… Since I had some issues with on former installation with a really huge Home Assistant Database, I didn’t care about the past and deleted it. It was only about 200 MB or so. So not much help from that.

So I dug a little deeper:

 _    _                                         _     _              _   
| |  | |                          /\           (_)   | |            | |  
| |__| | ___  _ __ ___   ___     /  \   ___ ___ _ ___| |_ __ _ _ __ | |_ 
|  __  |/ _ \| '_ ` _ \ / _ \   / /\ \ / __/ __| / __| __/ _` | '_ \| __|
| |  | | (_) | | | | | |  __/  / ____ \\__ \__ \ \__ \ || (_| | | | | |_ 
|_|  |_|\___/|_| |_| |_|\___| /_/    \_\___/___/_|___/\__\__,_|_| |_|\__|
                                                                         

Our command line:
$ ha help

~ $ 
~ $ pwd
/root
~ $ cd /
/ $ df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                   9.1G      8.1G    583.7M  93% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                   990.2M         0    990.2M   0% /sys/fs/cgroup
shm                      64.0M      8.0K     64.0M   0% /dev/shm
/dev/sda8                 9.1G      8.1G    583.7M  93% /config
/dev/sda8                 9.1G      8.1G    583.7M  93% /ssl
/dev/sda8                 9.1G      8.1G    583.7M  93% /addons
/dev/sda8                 9.1G      8.1G    583.7M  93% /media
/dev/sda8                 9.1G      8.1G    583.7M  93% /data
/dev/sda8                 9.1G      8.1G    583.7M  93% /backup
/dev/sda8                 9.1G      8.1G    583.7M  93% /share
tmpfs                   990.2M   1008.0K    989.2M   0% /run/dbus
/dev/sda8                 9.1G      8.1G    583.7M  93% /run/audio
/dev/sda8                 9.1G      8.1G    583.7M  93% /etc/asound.conf
/dev/sda8                 9.1G      8.1G    583.7M  93% /etc/hostname
/dev/sda8                 9.1G      8.1G    583.7M  93% /etc/resolv.conf
/dev/sda8                 9.1G      8.1G    583.7M  93% /etc/hosts
/dev/sda8                 9.1G      8.1G    583.7M  93% /etc/pulse/client.conf
tmpfs                   990.2M         0    990.2M   0% /proc/asound
tmpfs                   990.2M         0    990.2M   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                   990.2M         0    990.2M   0% /proc/scsi
tmpfs                   990.2M         0    990.2M   0% /sys/firmware
/ $

Ok, 8.1GB used disk-space they have to be somewhere right?

/ $ du -h -d 1
145.8M	./usr
4.6M	./lib
4.0K	./opt
4.0K	./srv
4.0K	./media
224.0K	./run
6.7M	./bin
4.0K	./tmp
8.0K	./root
4.0K	./home
0	./proc
80.0K	./var
0	./sys
4.0K	./mnt
232.0K	./sbin
2.1M	./etc
8.0K	./dev
8.0K	./share
123.5M	./config
20.1M	./backup
52.0K	./data
4.0K	./ssl
4.0K	./addons
92.0K	./libexec
303.5M	.

So 300 MB is not too much, and it seems I am inside a container, right?

Now my questions:

Are there any processes outside of the (for me) visible filestructure which are runnig wild (and consuming 8GB)?
How to access the host on shell-level?
I’d like to stick with the VM on the Synology. I am able to extend the virtual disk. How to extend the filesystem without access to the docker-host?

aceindy · October 18, 2020, 7:07pm

You might want to check any stale docking containers (you can use portainer), i found a couple of gigs there

markuskuttke · October 19, 2020, 12:39am

Yep, thanks. I found 7GB in images (not containers). Two could be removed. The others were all used by running containers. It freed 1GB and I could update.
Anyways, what is strange: under “Home” is see 18 running containers and 2 stopped. In the dashboard only 2 running and no stopped. What did I miss?
I’ll try to extend the VHDD now.

markuskuttke · October 19, 2020, 2:21am

Short update and (for now) satisfactory solution:

Expanded VHDD in Synology Virtual Machine Manager to 60GB
Installed SSH & Web Terminal from the Community Addon Store
Added prated, e2fsprogs, e2fsprogs-extra (last one contains resize2fs)

ssh:
  username: hassio
  password: password
  authorized_keys: []
  sftp: false
  compatibility_mode: false
  allow_agent_forwarding: false
  allow_remote_port_forwarding: false
  allow_tcp_forwarding: false
zsh: true
share_sessions: false
packages:
  - parted
  - e2fsprogs
  - e2fsprogs-extra
init_commands: []

Expanded /dev/sda8

resize2fs /dev/sda8

Done

➜  ~ df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  58.4G      7.0G     48.9G  13% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                   990.2M         0    990.2M   0% /sys/fs/cgroup
shm                      64.0M     68.0K     63.9M   0% /dev/shm
/dev/root                99.9M     99.9M         0 100% /sbin/docker-init
/dev/sda8                58.4G      7.0G     48.9G  13% /addons
/dev/sda8                58.4G      7.0G     48.9G  13% /backup
/dev/sda8                58.4G      7.0G     48.9G  13% /media
/dev/sda8                58.4G      7.0G     48.9G  13% /data
/dev/sda8                58.4G      7.0G     48.9G  13% /ssl
/dev/sda8                58.4G      7.0G     48.9G  13% /config
/dev/sda8                58.4G      7.0G     48.9G  13% /share
tmpfs                   990.2M      1.0M    989.2M   0% /run/dbus
/dev/sda8                58.4G      7.0G     48.9G  13% /run/audio
tmpfs                   990.2M      1.0M    989.2M   0% /run/docker.sock
/dev/sda8                58.4G      7.0G     48.9G  13% /etc/asound.conf
/dev/sda8                58.4G      7.0G     48.9G  13% /etc/hostname
/dev/sda8                58.4G      7.0G     48.9G  13% /etc/resolv.conf
/dev/sda8                58.4G      7.0G     48.9G  13% /etc/hosts
/dev/sda8                58.4G      7.0G     48.9G  13% /etc/pulse/client.conf

pietervanh · July 11, 2023, 11:09am

Thank you this helped me with an issue I had where my storage also was going away.
70 GB of Docker Images could be removed!
Supervisor should remove old Images and definitely remove images when you uninstall an addon.