Full Snapshots missing loads of files

mitchell · May 31, 2020, 11:15am

I just had to restore from a snapshot, and discovered that snapshots are not backing up all of the files.

the most notable for me was Scripts.yaml this has not been present in any of my full snapshots for at least a month, lucky, i had another back-up of my VM so i could restore from there

Check your Snapshots!

sublevel7 · May 31, 2020, 12:30pm

I can confirm that the problem exists with full snapshots triggered by a automation.

A manual full snapshot has more files (in the homeassistant.tar.gz-part or the complete tar-file). Left manual, right automatic:

petro · May 31, 2020, 12:33pm

@mitchell Mine are doing everything, seems like an issue with your automation. What are you using to perform the automated snapshots?

sublevel7 · May 31, 2020, 12:48pm

Can’t answer for @mitchell, but I think he is doing something like I do:

- id: Backup
  alias: Daily Backup
  trigger:
  - platform: time
    at: 03:00:00
  action:
  - service: hassio.snapshot_full
    data_template:
      name: Backup_{{ now().strftime('%d-%m-%Y') }}

greengolfer · May 31, 2020, 2:57pm

I had the same problem. I think I found the culprit but without solution…
I am using the default sqllite database so I have an “home_assistant_v2.db” file.
When the file is updated, it creates temporary files ending with .shm and .wal. You can see these files appearing and going away a few seconds later.
If the files are there when the snapshot is launched, then it will fail because, .shm and .wal files are missing (“normal” they are only temporary ones). The log:

20-05-10 11:58:49 INFO (SyncWorker_6) [supervisor.snapshots.snapshot] Snapshot folder share done
20-05-10 11:59:12 WARNING (SyncWorker_7) [supervisor.snapshots.snapshot] Can't snapshot folder homeassistant: [Errno 2] No such file or directory: '/data/homeassistant/home-assistant_v2.db-shm'
20-05-10 11:59:12 INFO (MainThread) [supervisor.snapshots] Full-Snapshot 25ec3bff done

So the snapshot is “done” but not complete…

GV

mitchell · May 31, 2020, 7:54pm

This is exactly the automation i am using I am not at my PC so I can’t test if a manual full backup does the same.

If a manual full backup works maybe triggering the backup via node red might be temporary work around?

I’ll investigate later.

If didn’t have that VM Disk backup I’d be really screwed, I can’t remember even half the stuff I have written for scripts.

I am very meticulous about my backups, I even make an image copy of all my PCs every night. I keep about 2 months of snapshots for HA and a weekly VM Disk backup in case the SSD fails.

I also test backups work when i set them up, so this was working, I am certain. An update to HA must have caused this. Knowing i can’t rely on on snapshots is very scary.

micque · May 31, 2020, 8:39pm

I remember there was an issue with snapshot and the database files causing incomplete backups about a year ago. I seem to recall a fix was put in place to resolve the problem. I guess it’s back?

greengolfer · June 1, 2020, 6:15am

Unfortunately not. I already tried that. Same behaviour.

It is not always failing. If you are lucky, that temporary files do not appear while doing the backup and you are fine. It is (I strongly think) a matter of timing.

Agreed. I am using proxmox. And save the VMs weekly. Once I add to get back to it, as some “big” files after restore were missing.

If you search on the forum you will see various posts with the same issue. Never saw a fix for it.

GV

mitchell · June 1, 2020, 8:51am

I am on unraid, so i think i can set up a script to copy the files via the SMB share. But this is extra work for unraid, and a bit messy, but might be the only option for now.

greengolfer · June 1, 2020, 9:07am

Yep… In my case, as I have a synology nas, I use use to backup/save the entire HA configuration. The backup is initiated by the NAS. So, no risk to loose files. As least, so far it didn’t happen!!

GV

sublevel7 · June 1, 2020, 5:32pm

I’ve added the config-folder to my nightly backup on the Synology, too. Until now I used Active Backup just to get the snapshot-files to a second location.

But I see this only as a workaround. A backup should be working bullet-proof.

I googled the term from the error log at github and found, that there is a error alreay reported at 18.10.2018 in issue #17583, later merged to issue #779 and still open.

greengolfer · June 1, 2020, 5:41pm

Agreed. It seems to be a rather basic stuff and… nothing.
Weird.

GV

francisp · June 1, 2020, 5:53pm

Well, I use googledrivebackup ( an add-on for home assistant complete/ home assistant supervised) and my backups are reliable. Restore is something different Friday I tried to restore on a new installation, at the first attempt only the add-ons were restored, nothing of the configuration. Only after I restored the same backup again everything was there.

seanomat · June 3, 2020, 6:48am

I solved this problem by moving the database files to the “shared” folder.
This way the missing database file doesn’t corrupt the backup of the config folder.
There is a long thread on GitHub for this issue.

jdbrookes · June 4, 2020, 10:57am

Can you briefly confirm how you moved the database to a different folder? Would the formatting be as below:

recorder:
  purge_keep_days: 5
  db_url: sqlite:////home/user/.homeassistant/share/database

Where “database” is a folder I created in the share folder, visible when browsing via samba. Did you move the db file before or after restarting HASSIO?

I’ve just checked several of my automated Google Drive snapshots and have been horrified to find that half of the configuration has not been backed up.

seanomat · June 4, 2020, 1:26pm

This is the path I used:

sqlite:////share/hassio_db/home-assistant_v2.db

To retain the data, I connected via ssh, stopped home assistant, copied the files, started home assistant again. After verification I deleted the old files.
If you don’t need to retain the data, you can just change the path and do a restart. The file will be created, if it doesn’t exist.
I was shocked as you to find all of my backups useless, and astonished, that obviously this is not regarded as top priority issue.

jdbrookes · June 4, 2020, 3:20pm

Thanks, I did exactly that just now and it seems to be working perfectly in its new location.

For anyone wondering:

Add the following to your config under recorder: db_url: sqlite:////share/hassio_db/home-assistant_v2.db
Create a folder “hassio_db” in the share folder of your hassio install
Connect via SSH, enter command ha core stop
Copy the database file to the new location hassio_db
Enter SSH command ha core start and you’re back in action. Old database file can be deleted then

Hope this issue is resolved as a priority on github

mitchell · June 4, 2020, 3:56pm

I can just redirect it without moving if I do not care about the data?

francisp · June 4, 2020, 3:59pm

Yes you can.

Arjen_W · June 5, 2020, 7:13am

I noticed the same and the solution provided here as listed by @jdbrookes worked for me.

I agree that this should be a high priority issue. Maybe if we all comment on the Github issue, it might get attention of the devs.