Running out of space - what can I delete to create a new backup?

typxxi · July 6, 2023, 3:03pm

I have set up a new rasperry pi 3 with a 8 GB SD Card 6 weeks ago and have played a bit with the new solar power features and what not.
Now I wanted to prepate the move to a SSD which of cause first requires a backup of the current system

But the scheduled backup has not been working and I get the warning: Low Space 0.2 GB

I am running Hassio 2023.5 and could also not make an update I guess due to the missing backup space.
There is no current backup on the sd card anymore (the monthly backup had been moved to a NAS already)

my config folder shows a huge database

total 430032
-rw-r--r--    1 root     root          6302 Jun 11 23:54 automations.yaml
drwxr-xr-x    4 root     root          4096 May 16 21:10 blueprints
-rw-r--r--    1 root     root          2196 Jul  6 12:21 configuration.yaml
drwxr-xr-x    8 root     root          4096 Jul  6 11:37 custom_components
drwxr-xr-x    2 root     root          4096 May 16 21:10 deps
-rw-r--r--    1 root     root             0 Jul  6 16:46 files.txt
-rw-r--r--    1 root     root         72977 Jul  6 16:41 home-assistant.log
-rw-r--r--    1 root     root       1713314 Jul  6 11:38 home-assistant.log.1
-rw-r--r--    1 root     root             0 Jul  6 11:39 home-assistant.log.fault
-rw-r--r--    1 root     root     433594368 Jul  6 16:45 home-assistant_v2.db
-rw-r--r--    1 root     root         32768 Jul  6 16:46 home-assistant_v2.db-shm
-rw-r--r--    1 root     root       4873992 Jul  6 16:46 home-assistant_v2.db-wal
drwxr-xr-x    2 root     root          4096 May 18 10:11 pyscript
-rw-r--r--    1 root     root             0 May 16 21:10 scenes.yaml
-rw-r--r--    1 root     root           294 Jul  6 11:59 scripts.yaml
-rw-r--r--    1 root     root           161 May 16 21:10 secrets.yaml
drwxr-xr-x    2 root     root          4096 Jul  6 11:54 shell
drwxr-xr-x    2 root     root          4096 May 19 05:07 tts

How can i reduce the size of the home-assistant_v2.db or where else can I delete tmp files ?

I have searched the help and found a solution that should clean the log from 2018 which did not work out and then I checked the provided links for what to do in case of low space but I did not understand what to do and how

Sorry, but maybe I missed a menu cause I can not find such maintenance menu and I need a backup first before I can move ahead to a far bigger SSD.

thanks a lot

tom_l · July 6, 2023, 3:17pm

See: Database Restore after Snapshot restore - #2 by tom_l

You can delete the database while home assistant is stopped and when you start it a new empty one will be created.

If you don’t care about the history you can skip copying the database.

So: stop, delete, start.

Arh · July 6, 2023, 3:49pm

Can you not just plug in a usb drive and backup to that after setting it up as storage? Or share a folder on your pc and set that as backup storage? I am just guessing you could do this as I have never tried either.

typxxi · July 6, 2023, 4:07pm

no, afaik the backup is created locally and will be moved afterwards to the NAS.
So I need to free space first, then backup and finally migrate to the ssd.

I will try the idea tom_i had mentioned above.

typxxi · July 6, 2023, 5:19pm

I opened my terminal in hassio and then typed

ha core stop

as you had mentioned in the link.

Within a few seconds I was screwed:
I had no more access via browser to the terminal or what so ever to type in my commands.

I have started a copy of the config files to the NAS. Will take quite some time.
But how can I delete the db file while ha is running to get the space I will need for the backup or are these files not protected while HA is running?

#1 UPDATE: I was able to delete the db file in the /config/ folder

sudo rm home-assistant_v2.db

#2 UPDATE: 0,6 GB free and still not enough for backup according to superuser log

Any ideas what to delete next ? I have not installed many repositories.
Is there a cli command for listing all files / folders sorted by size ?

CaptTom · July 6, 2023, 6:53pm

Have you looked in the backup directory? It’s at the same level as the config directory. Not sure what else could be eating up 8G on an RPi with just HA on it and nothing else.

typxxi · July 6, 2023, 7:15pm

of cause: it is an empty dir

Is there any command to sort all files by size ?

ncdu is not available

typxxi · July 6, 2023, 9:59pm

STATUS QUO:

I have found a filesize check here

which explains a bit more than I will explain here. Be aware that the syntax over there has changed a bit since at least regarding folder / file depth which is not “-d 1”.

du -h -d 1 /

Result is this, which I have manually formatted and resorted to make it more readable

1400M /
 569M /media
 483M /var
 313M /usr
  44M /config 
  13M /package
  11M /root
   7M /lib
   3M /bin
   2M /etc
   1M /sbin
   1M /run
  76K /tmp
  60K /data
  52K /command
  12K /home 
   4K /opt
   4K /mnt
   4K /srv
   4K /ssl
   4K /addons
   4K /share
   4K /backup
   4K /lib64
   0  /sys
   0  /proc
   0  /dev

Strange thing: if you add it up you will have a sum of just 2.847 MB but I have a 8 GB SD card and still just 0,6 GB free and I am not able to run a backup.
Yes, i have restarted the system 2 times fully.

What is wrong ?
And what about 1,4 GB of files in the root directory I can not see?

tom_l · July 7, 2023, 1:21am

Hassio has not been a thing since 2020. If you really did that you are using a very old version.

Assuming you meant HA OS, the ssh/terminal addons run in their own containers and will not be affected by stopping home assistant (they keep running). Unless you access the terminal via the home assistant front end. Then when that frontend shuts down you will lose access via that method. Apologies I thought that it would be obvious not to use a program you are shutting down to access said program.

Use an SSH application on an alternate PC for access.

Another method for creating a backup that will not require any free disk space is to copy the entire /config folder, including all hidden folders within it, off the SD card. This can be copied to a new system to restore. You will then only have to set up any addons you use. To speed this up you can copy/paste the addon configuration pages to a temporary text file in yaml format off system.

reotto · July 7, 2023, 1:36am

Install Samba and then install Samba backup and you can backup externally.

tom_l · July 7, 2023, 1:38am

As mentioned above, that won’t work. Backups are created locally before being moved off system.

reotto · July 7, 2023, 1:42am

Well shoot I never knew that. My apologies.

paddy0174 · July 7, 2023, 2:17am

Don’t you have another computer at hand? A Windows or Linux machine will do, eg. your laptop?

Look for a cloning program to clone the SDcard to a bigger (new) one. That should work easily and you can, depending on how the program works, later resize the partitions or this is already done while cloning.

Should take you not more than half an hour, here depending on the speed your SDcard reader in the laptop has.

Afterwards put the card back in and run it like normal.

EDIT: someting like this (it’s the first result for a google search, so no recommendation!)

typxxi · July 7, 2023, 6:11am

Yes, of cause but the whole story was, that I needed a backup only to migrate to the bigger SSD.
That way I would have to get a bigger SD card first, clone that to an image on my Computer, then image back to bigger SD, boot HA from that again to be finally able to backup the system.

That would be my ultimate solution I will prepare.

Not so short, but maybe shorter then to find the directory to delete another 200 MB to run a usual backup again.
But thanks, I will try that later.

typxxi · July 7, 2023, 8:08am

Yes and no,

yes, cause finally it worked out, but it took quite a while cause my usual Win32DiskImager did not wanna create an image of this 8 GB SD card for unknown reasons.

Then I have used another free tool to be able to create an image file on a harddrive too which can be found here from passmark software

and remember the no from above, I mean the yes and no ?

Not sure if you have to resize the sd cause when I bootet that new 32 GB SD for the first time it took ages - about 5 minutes or so till HA reappeared. It felt like a lot was going on and part of that might have been an automatic resizing cause once booted the system - storage showed 21 GB free of 32 GB in total.

So problem solved but I will write all of my steps together later that others might not have to fight so long as I had to fight. Right now I am upgrading first and then will start a real HA backup to finally move ahead to the Pi 3 booting from SSD solution which is another topic of cause.

thanks a lot

tom_l · July 7, 2023, 8:35am

Hopefully not many others will be in this situation by taking note of the recommended minimum storage (32GB).

typxxi · July 7, 2023, 11:30am

Yes, I had read that, but it was a fresh and clean install to be used as a sandpit to play around. I did not have the SSD when I started and had to wait for all 5 SSD that I had ordered for my Pi’s and went missing during shipping.
I had only mutliple 8 GB SD cards and a 128 GB for the drone footage which I needed there.
So I had started with that, but I guess there are more cause I found a few that had similiar issues but were able to delete more backups and what not.
I had started with 1,8 GB free storage and those were gone within 6 weeks which I had not seen before even though I have to say that I had not tracked energy consumption and solar power production with HA before and I bet the 5 inverters will have caused a lot of data.

And unfortunatelly I had not understood how to purge a bit of the recording, therefore had hoped for some what temp files someone would new to delete to get enough free storage for further backups.

Now it is running fine on a 32 GB SD Card my friend had brought along.
Backup and upgrades worked fine, hope the migration from SD to SSD for the Pi 3+ will work as described in this video which will start for the Pi 3+, but shows in the beginning also the Pi 4 migration which looks to be a bit easier.

Thanks a lot @tom_l for your great and fast support so far !

typxxi · July 8, 2023, 10:28am

I guess I have found my real issue.

I had used the terminal in HA OS to check for big files and what not assuming I would be looking at the full file system

But I guess that HA OS means that a lot of space is used by HA OS which I can only check via ssh with putty cause only then I will see the full file system. I could have lost space there too by those HA updates I had done before or whatever.

At least for now I have plenty of space since the 128 GB SSD is running now.