Automated snapshot optimisation

Thought this might be of use to someone.

I don’t use the google drive backup addon as I don’t like using passwords on my snapshots (as it is impossible to extract files without restoring) and do not want this unprotected data in the cloud.

So I create an automated snapshot every night and use a file sync utility on another always on PC to copy the snapshot. This is then copied to my off-site backup monthly.

Later in the morning I run the snapshot cleanup service to only keep the last 20 snapshots.

The problem is that my snapshots have become huge (820MB+) and while they do complete successfully they do generate time out / api errors. This is despite the liberal use of the exclude option in the recorder.

So instead of doing a full snapshot I now do a partial snapshot of everything except the MariaDB and ESPhome addons. I can live with lost history data and all the esphome files required get backed up in the config folder anyway.

Also I have the MariaDB config saved to a text file for easy restoration.

This has taken my backups from 7 minutes of 70% peak processor use generating an 820MB+ file to 10s of seconds at 20% CPU generating a 60MB file.


There is not a lot of information on how to automate a partial snapshot so I include my automation here (with the old full snapshot config commented out):

- id: daily_backup_at_1_am
  alias: Daily Backup at 1 AM
  initial_state: true
  trigger:
    platform: time
    at: '01:15:00'
  action:
  # - service: hassio.snapshot_full
  #   data_template:
  #     name: Automated Backup {{ now().strftime('%Y-%m-%d') }}
  - service: hassio.snapshot_partial
    data_template:
      name: "Automated Backup {{ now().strftime('%Y-%m-%d') }}"
      addons:
        # - 15ef4d2f_esphome
        - a0d7b954_aircast
        - a0d7b954_portainer
        - a0d7b954_ssh
        - a0d7b954_vscode
        - addons_local
        - core_duckdns
        # - core_mariadb
        - core_mosquitto
        - core_samba
        - core_ssh
      folders:
        - homeassistant
        - share
        - ssl
  - service: notify.telegram_system
    data_template:
      title: '*Information*'
      message: "Creating backup named Automated Backup {{ now().strftime('%Y-%m-%d') }}"

The names of your addons can be found inside a full snapshot or by looking at the URL when viewing the hassio addon page for each addon.

I don’t know what addons_local is but it was in the full snapshot so I included it (it’s empty with my selection of addons). Interestingly it does not show up in the partial snapshot (right hand side).

I just noticed the VSCode addon is responsible for over three quarters of the snapshot size now and as it has the default config and is easy to replace I’ll remove that from the snapshot too.

5 Likes

According to the snapshot tab in the “supervisor” section, addons_local is a folder and not an addon. So maybe you should adapt your automation.

It would be nice to add a doc section to home-assistant.io website regarding snapshots and their configuration.

I’ve already removed it as I was getting warnings in the system log.

@tom_l, I’ve noticed that ESPHome takes way more space than it should on my snapshots too. I prefer to back up the yaml files from the ESPHome directory but I’d like the snapshot function to do that. I just posted the detail about it here: Snapshot size dramatically increased

Anyway, the ESPHome archive size doesn’t seem to follow the config/esphome directory size unless I’m missing something big somewhere. I’d like to solve that problem without excluding the ESPHome yaml files.

That’s because of the way small files are stored, see:

Hi @tom_l, I’m guessing you didn’t see my screen shots:

image

Size on disk: 6.38MB

esphome.tar.gz: 34.1MB

Usually, the tarred and zipped stuff should compress out nicely and not take up more space than it does on the disk due to sector size. What am I missing?

The esphome tar.gz file is the addon container and program, not the device configs.

Look at the size of the homeassistant tar file. That contains your config folder and esphome configs. Look inside it.

1 Like

OMG, you’re right! What the heck do I want to save off gigantic versions of ESPHome for??

Thanks for sharing this,… I’d been looking for some documentation on snapshot and how to ‘trim’ my snapshots,… as they were over 10gigs each,… ( due to cam video stored in the shared folder ),… Now I have a nice <50Megs per snapshot,… and much more manageable.
Thankyou

1 Like