Home assistant backup folder repeat itself taking up a lot of space

kavierkoo · May 28, 2020, 12:40pm

Hey all!!

I am running HA on Docker, I configure it to backup daily and recently I notice that it is taking up 3GB for just 1 day.

After i SAMBA into it and take a look, it looks like this

and I can just keep clicking into the backup folder, it just seems to go on and on, like this

so… is it my configuration issue or it is suppose to look like that?
I just removed some old snapshot to free up some space, (7 days of snapshot was 22GB)

Appreciate someone can shed me some light!!

Updated with more information below:

HA Info:
HA Version: 0.110.3
OS: Ubuntu
Supervisor version: 224
Autobackup version by @jcwillox: 0.5.2

Configuration.yaml:

recorder:
  purge_keep_days: 1
  purge_interval: 1

auto_backup:
  auto_purge: true

Automation.yaml

- alias: Perform Daily Backup
  trigger:
    platform: time
    at: 05:00:00
  action:
    service: auto_backup.snapshot_full
    data_template:
      name: 'DailyBackup: {{ now().strftime(''%A, %B %-d, %Y'') }}'
      keep_days: 7

myle · May 29, 2020, 5:00am

what size is you database file

home-assistant_v2.db

mite have to do some reading on the recorder

my last 7 days

about 8gb

tom_l · May 29, 2020, 5:39am

That’s not the issue. This is the issue:

9740b6b5412edab3b2bd7cfc5bbf4a468fe92818

Each one of those folders contains a copy of the backup. Which is 90MB.

kavierkoo · May 29, 2020, 5:40am

Hi Myle,

My database size it 827.15 MB, i use autobackup to keep track of the snapshots.

Your screenshot of snapshot seems like how it should work instead of mine,
which the backup folder another backup folder which loop for a ridiculous amount of time ( as shown below) and eat up a large amount of storage

kavierkoo · May 29, 2020, 5:40am

Yep, this is the issue. thanks for clarification, it can actually still go on and on until it took up 3GB for 1 day

tom_l · May 29, 2020, 5:41am

It’s definitely not normal.

kavierkoo · May 29, 2020, 5:42am

worst thing is I have no idea where to start the troubleshooting… thats why im here to seek for some advises

myle · May 29, 2020, 6:04am

bugger
I use the google one

dum qusetion
what your config options

kavierkoo · May 29, 2020, 6:06am

my google drive running out of space sadly… haha

Here are my configs

Configuration.yaml:

recorder:
  purge_keep_days: 1
  purge_interval: 1

auto_backup:
  auto_purge: true

Automation.yaml

- alias: Perform Daily Backup
  trigger:
    platform: time
    at: 05:00:00
  action:
    service: auto_backup.snapshot_full
    data_template:
      name: 'DailyBackup: {{ now().strftime(''%A, %B %-d, %Y'') }}'
      keep_days: 7

jcwillox · May 29, 2020, 6:22am

That’s pretty broken, not sure how you got all those subdirectories, but there should only be *.tar files in the root /backup directory.
I don’t see any issue with your configuration.

I think it’s quite weird that every subdirectory is the same and that it appears to go on forever, so my guess is actually that somehow a directory junction (or hard link) has been used to create a loop. That is that /backup/backup actually links to /backup but the OS doesn’t perceive it that way and so it appears as infinite subdirectories.

To check this open the windows command prompt in the /backup directory and run the command dir, you looking for something saying <JUNCTION>.

You can try deleting anything that is not a *.tar file in the root /backup directory, and see if the problem persists (i.e. it’s still creating subfolders). It’s also worth checking the home assistant logs and supervisor logs around the time you take a snapshot. Are you running any other backup utilities that may be conflicting?

kavierkoo · May 29, 2020, 7:03am

Hi Josh,

Thanks for responding!
I’m not using any other backup utilities.

I tried to DIR and look for junction but didnt find anything

Although, I found that there’s a link after
docker exec -it hassio_supervisor bash

hmm this seems to be the issue!
After rm -rf it, and re-initiate another snapshot, the problem is gone!

To all:
Appreciate for everyone’s tips!

Kudos!

jcwillox · May 29, 2020, 7:31am

Great! so it was a circular link on the linux side, it’s weird that was even created, it couldn’t have been home assistant or any of its components, as they don’t have access to the /backup folder, so maybe the supervisor or an addon caused it.

kavierkoo · May 29, 2020, 7:35am

Hi @jcwillox ,

Now I have another issue, after deleting the backup folders,
Supervisor > Snapshot still show there are 3 snapshots but in backup directory there’s only 1

and Autobackup shows even more

I tried to click on Supervisor>snapshot>delete, nothing happened.

Would appreciate if anyone can advice on how do I delete these records?

jcwillox · May 29, 2020, 9:15am

Auto Backup shows a snapshot as monitored until it expires even if it’s deleted externally, you could manually edit/delete the file under /config/.storage/auto_backup.snapshot_expiry and restart.
As for the supervisor it creates the list of snapshots by reading from the /backup directory. There should be a reload button in the top right corner, otherwise a restart should fix it.

kavierkoo · May 29, 2020, 9:53am

Hi Josh,

Awesome, it solved all the problem after deleting the auto_backup.snapshot_expiry and restart!

Million thanks!
kudos!

sota · May 29, 2020, 1:46pm

@kavierkoo I had a similar problem recently but it was with my config folder. I only discovered it after the Visual Studio Code plugin started pegging my i5 NUC CPU useage at 100%. Turned out that it was disappearing down the infinite plughole of the config folder enumerating files as it went! I couldn’t get rm -rf to remove the link, so I exported a backup and used 7-zip to delete it. Once I did a wipe & restore, it was fine.

However, on checking today I find that my backup folder has the same problem as you, but this time the rm -rf worked, but it is a strange problem!

kavierkoo · May 29, 2020, 2:01pm

@sota yea lol… When I run hassio directly on Ubuntu, I never have this issue,
It only happen after I dockerize it…
Are you running on linux? or Docker?

Let’s see if we have any similarity here.
I’m running it on Docker, and Addons are:

AdGuard Home
SSH & Web Terminal
Samba Share
HACS
Auto Backup
and few of frontend cards.

I use automation to snapshot daily 5 AM and keep for 7 days, and uses auto backup more insight info.

sota · May 29, 2020, 3:25pm

I’m running HA Supervised under Ubuntu Server 18.04 along with about 10 or 12 other containers under docker compose.
I noticed the problem because the NUCs fan was running flat-out after installing VS Code. Other than that, I have:
Appdaemon 4
ESPHome
Google Drive Backup which runs every night
Let’s Encrypt
MariaDB
Mosquitto Broker
Node-Red
SSH & Web Terminal
Samba
Zigbee2mqtt

I suspect that file corruption may be at the root of the problem. However, it’s almost impossible to prove so I guess I’ll just keep an eye on it for now. I am planning to move to Proxmox, so this may force me to get on with it.

kavierkoo · May 30, 2020, 5:57am

I see, I noticed mine after seeing the HDD just keep increasing on lovelace dashboard,

Hmm, the common thing I guess was we both running HA Supervised with Docker on the same OS and version too.

and yea, we can only monitor it for now. Good info about Proxmox… but I think the dashboard not as elegant compare to HA (just my opinion)… I do use Netdata and customize my own small dashboard to monitor my webserver tho.

(Still having trouble with Netdata on docker not able to collect logs from HDDTEMP)

kavierkoo · October 25, 2020, 6:02am

Ugh!! the problem resurfaced, this time not only backup folder having the issue, all other folders are looping as well.

addons
backup
config
share
ssl

all folders are looping.

I have to do below to remove the links:

docker exec -it hassio_supervisor bash
find /data -type l
unlink each of them
ctrl + p + q
docker exec -it homeassistant bash
unlink /config/config