Backups failing, coincident with 2025.10.0 update

jeffsf · October 9, 2025, 12:31am

I recently brought up HAOS on an ODROID C2 and have found it to be quite useful. I was able to configure both local backups as well as backups using the WebDav integration to self-managed Nextcloud. Several ran sucessfully. I can see the initial few backups on my Nextcloud server and they sync from there to my Mac as I had hoped.

I am preparing to move to different hardware and went to check the backups and the “Creating Backup” and “Backup now” animations are spinning. This persists until I restart Home Assistant. After the first few successes, the backups are not appearing on NextCloud.

I have tried removing and re-adding the WebDav account and the situation reoccurs when I start a backup. I can see successful PROPFIND and GET requests from HomeAssistant in the logs.

Curiously, I just noticed that the change in behavior seems coincident with upgrading to 10.0 and continued through 10.1 (see image).

2025-10-04 08:55:39.376 INFO (MainThread) [supervisor.backups.manager] Backup 1ac984d9 starting stage addon_repositories
[...]
2025-10-04 08:56:29.313 INFO (MainThread) [supervisor.backups.manager] Creating partial backup with slug 1ac984d9 completed

so it appears that the backups ran successfully.

Yesterday, I removed the WebDav destination and the backups fail to the local destination (which should be the only destination). Click-initiated backups, either “Automatic” or “Manual” (with only the local destination selected) do not complete.

Is this a known issue?

Until I get this resolved, what are the directories that I need to tar up or rsync to transfer this install to new hardware?

Thanks!

➜  ~ ls -Al /backup
total 1091148
-rw-r--r--    1 root     root       1495040 Sep 28 19:38 2025-09-28-before-HACS_2025-09-28_19.38_27079635.tar
-rw-r--r--    1 root     root      97464320 Oct  1 20:14 2abd07f8.tar
-rw-r--r--    1 root     root     106188800 Oct  2 04:55 Automatic_backup_2025.10.0_2025-10-02_04.55_06003221.tar
-rw-r--r--    1 root     root     167680000 Oct  4 05:31 Automatic_backup_2025.10.1_2025-10-04_05.30_34002668.tar
-rw-r--r--    1 root     root     171919360 Oct  4 08:56 Automatic_backup_2025.10.1_2025-10-04_08.55_39366665.tar
-rw-r--r--    1 root     root     196792320 Oct  5 05:28 Automatic_backup_2025.10.1_2025-10-05_05.27_17003250.tar
-rw-r--r--    1 root     root     227031040 Oct  6 05:43 Automatic_backup_2025.10.1_2025-10-06_05.42_27003272.tar
-rw-r--r--    1 root     root      25487360 Sep 29 04:48 Automatic_backup_2025.9.4_2025-09-29_04.48_36002798.tar
-rw-r--r--    1 root     root      48558080 Sep 30 05:07 Automatic_backup_2025.9.4_2025-09-30_05.06_57003200.tar
-rw-r--r--    1 root     root      74700800 Oct  1 05:01 Automatic_backup_2025.9.4_2025-10-01_05.01_08003238.tar

hassos: “16.2”
homeassistant: 2025.10.1

For add-ins, all I believe I have added is the Advanced SSH & Web Terminal.

francisp · October 9, 2025, 4:10am

/config and everything beneath, including hidden files and directories

does not include add-ons, but Advanced SSH & Web Terminal has no important things to backup

Sir_Goodenough · October 9, 2025, 3:08pm

Except remember your login and password from the configuration…

jeffsf · October 23, 2025, 3:30am

Investigation shows the local backup files are present and appear to be complete. The jobs shown by ha jobs indicate that they completed successfully in creating the local backup. The WebDAV destination never gets a copy. The logs show that the HA Core never logs the “backup complete” message.

Filed

github.com/home-assistant/core

Backups fail to signal completion

opened 03:07AM - 23 Oct 25 UTC

jeffsf

### The problem I started noticing problems with GUI-initiated backups failing …to store to a WebDAV destination and "spinning forever" perhaps coincidentally with upgrade to 10.0. Backups even to local destinations began to spin forever shortly thereafter. Problem stated at https://community.home-assistant.io/t/backups-failing-coincident-with-2025-10-0-update/938487 with no solution. I have since transitioned to other hardware and the backups do not complete from the perspective of the GUI and recorded states. They show as failed even though the tar files are present on the local media. Claude pulled together the analysis. I'll spare you its suggestions as I don't know the codebase well enough to know if it is hallucinating or being truly useful. ## Summary Backups fail to complete on HAOS installations. Home Assistant Core initiates a backup successfully and receives initial progress events, but the backup process hangs indefinitely after the `addon_repositories` stage. The backup job completes successfully in Supervisor, but Home Assistant Core never receives the completion event. ## Environment Installation method: Home Assistant OS Core: 2025.10.3 Supervisor: 2025.10.0 Operating System: 16.2 Frontend: 20251001.4 ``` # ha info arch: amd64 channel: stable docker: 28.3.3 features: - reboot - shutdown - services - network - hostname - timedate - os_agent - haos - resolved - journal - disk - mount hassos: "16.2" homeassistant: 2025.10.3 hostname: homeassistant logging: info machine: qemux86-64 machine_id: 72b21e07527745b48ed954584c746819 operating_system: Home Assistant OS 16.2 state: running supervisor: 2025.10.0 supported: true supported_arch: - amd64 - i386 timezone: America/Los_Angeles ``` Both bare metal on ODRIOD C2 and Proxmox VM running OVA image on Dell Wyze 5070 (Pentium 5005) have been impacted. ## Steps to Reproduce 1. Initiate a backup through Home Assistant UI 2. Observe backup progress in logs 3. Backup progresses to `addon_repositories` stage 4. No further progress is shown 5. Backup appears to hang indefinitely ## Expected Behavior 1. Backup should progress through all stages 2. Backup should complete successfully and report completion to user 3. Backup file should be available in configured locations ## Actual Behavior 1. Backup reaches `addon_repositories` stage 2. No further progress events are received 3. Home Assistant Core remains in "creating backup" state indefinitely 4. User interface shows backup as "in progress" forever 5. Backup file IS created successfully locally 6. Backup is NOT written to the WebDAV store, if enabled 7. Supervisor job shows as completed successfully ## Evidence ### Home Assistant Core Logs ``` 2025-10-22 16:23:12.259 DEBUG (MainThread) [homeassistant.components.backup] Backup state: idle -> create_backup 2025-10-22 16:23:12.259 DEBUG (MainThread) [homeassistant.components.backup] Received backup event: CreateBackupEvent(manager_state=<BackupManagerState.CREATE_BACKUP: 'create_backup'>, reason=None, stage=None, state=<CreateBackupState.IN_PROGRESS: 'in_progress'>) 2025-10-22 16:23:12.261 DEBUG (MainThread) [homeassistant.components.backup] Finished fetching backup data in 0.000 seconds (success: True) 2025-10-22 16:23:12.296 DEBUG (MainThread) [homeassistant.components.backup] Received backup event: CreateBackupEvent(manager_state=<BackupManagerState.CREATE_BACKUP: 'create_backup'>, reason=None, stage=<CreateBackupStage.ADDON_REPOSITORIES: 'addon_repositories'>, state=<CreateBackupState.IN_PROGRESS: 'in_progress'>) 2025-10-22 16:23:12.296 DEBUG (MainThread) [homeassistant.components.backup] Finished fetching backup data in 0.000 seconds (success: True) ``` (No further log entries related to backup) ### Supervisor Job Status Output from `ha jobs info` for the backup job: ```yaml ignore_conditions: [] jobs: - child_jobs: - child_jobs: [] created: "2025-10-22T23:23:12.278812+00:00" done: true errors: [] extra: null name: backup_store_homeassistant progress: 100 reference: "21294352" stage: null uuid: e8b4bf941bad44e598e561d36c06ea02 - child_jobs: - child_jobs: [] created: "2025-10-22T23:23:34.687711+00:00" done: true errors: [] extra: null name: backup_addon_save progress: 100 reference: a0d7b954_ssh stage: null uuid: 84f1b872c8d4469d8ea7c3b9ed0daa2c created: "2025-10-22T23:23:34.687558+00:00" done: true errors: [] extra: null name: backup_store_addons progress: 100 reference: "21294352" stage: null uuid: 4d3d2b6acfb64d2398e949541dedd9b4 - child_jobs: - child_jobs: [] created: "2025-10-22T23:23:34.906561+00:00" done: true errors: [] extra: null name: backup_folder_save progress: 100 reference: ssl stage: null uuid: c5d70a43b6ad41f4b94fa83c36360362 created: "2025-10-22T23:23:34.906366+00:00" done: true errors: [] extra: null name: backup_store_folders progress: 100 reference: "21294352" stage: null uuid: 441f0106bd19469086a3074a6a281b21 created: "2025-10-22T23:23:12.269546+00:00" done: true errors: [] extra: null name: backup_manager_partial_backup progress: 100 reference: "21294352" stage: finishing_file uuid: 5b1c9ddfc7874a239463af988ac9b2db ``` ### Filesystem Evidence The actual backup file on disk confirms the timeline: ```bash # stat /mnt/data/supervisor/backup/manual-no-media-no-share-all-add-ons-both-destinations_2025-10-22_16.23_12261913.tar File: manual-no-media-no-share-all-add-ons-both-destinations_2025-10-22_16.23_12261913.tar Size: 782796800 Blocks: 1528912 IO Block: 4096 regular file Device: 811h/2065d Inode: 131102 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-22 16:23:12.275634462 -0700 Modify: 2025-10-22 16:23:34.918733057 -0700 Change: 2025-10-22 16:23:34.918733057 -0700 ``` ### Key Observations From Supervisor job status: - Job status: `done: true` - Job errors: `[]` (empty - no errors) - Job progress: `100` (complete) - Final stage: `finishing_file` - All child jobs completed successfully: - `backup_store_homeassistant` - done - `backup_store_addons` (including addon `a0d7b954_ssh`) - done - `backup_store_folders` (ssl folder) - done - Job creation time: `2025-10-22T23:23:12.269546+00:00` (16:23:12 local time) - Job completed by: `2025-10-22T23:23:34.906561+00:00` at latest (16:23:34 local time) - Total time: ~22 seconds from start to completion From filesystem: - ✅ File access time: `16:23:12.275` - matches when Supervisor job started (0.006 seconds after job creation) - ✅ File modify time: `16:23:34.918` - matches when Supervisor job completed (0.012 seconds after job marked done) - ✅ File size: 782,796,800 bytes (~746 MB) - backup is complete and valid - ✅ Write duration: 22.643 seconds from first access to final modification This ~~proves~~ _suggests:_ 1. The backup file was successfully created and completed 2. The completion timestamps precisely match between Supervisor job status and filesystem 3. The file is not corrupted or incomplete (full 746 MB file) 4. Supervisor successfully finished writing the backup file at 16:23:34.918 5. HA Core was still waiting for events at this time and beyond 6. Three independent sources (HA Core logs, Supervisor job status, filesystem) all corroborate the timeline ## Timeline Analysis Comparing Home Assistant Core logs with Supervisor job timestamps and filesystem evidence: | Time | Event | Source | |------|-------|--------| | 16:23:12.259 | Backup state: idle -> create_backup | HA Core | | 16:23:12.269 | Job created in Supervisor | Supervisor | | 16:23:12.275 | Backup file first accessed (write begins) | Filesystem | | 16:23:12.278 | backup_store_homeassistant started | Supervisor | | 16:23:12.296 | Last event received by HA Core (addon_repositories stage) | HA Core | | 16:23:34.687 | backup_store_addons completed | Supervisor | | 16:23:34.906 | backup_store_folders completed | Supervisor | | 16:23:34.906 | Job completed (stage: finishing_file) | Supervisor | | 16:23:34.918 | Backup file last modified (write complete) | Filesystem | | [never] | Completion event received by HA Core | ❌ Missing | **Critical Gap:** Supervisor completed the backup at 16:23:34 (~22 seconds after start), confirmed by filesystem write completion at 16:23:34.918, but HA Core never received any events after 16:23:12.296 (the initial addon_repositories event). During the 22+ seconds of backup activity, HA Core received zero progress updates. ## Technical Analysis ### Event Flow Architecture Home Assistant Core receives job progress events from Supervisor through the following mechanism: 1. **Supervisor → HA Core WebSocket Connection** - Supervisor sends events via websocket to HA Core - Endpoint: `websocket_supervisor_event` in `homeassistant/components/hassio/websocket_api.py:99-104` 2. **Event Dispatcher** - Events are dispatched internally via `async_dispatcher_send(hass, EVENT_SUPERVISOR_EVENT, msg[ATTR_DATA])` - Signal name: `EVENT_SUPERVISOR_EVENT = "supervisor_event"` 3. **Event Listener in Backup Code** - Located in `homeassistant/components/hassio/backup.py:714-739` - Method: `_async_listen_job_events()` - Subscribes to `EVENT_SUPERVISOR_EVENT` signal - Filters for matching job UUID 4. **Progress Callback** - Located in `homeassistant/components/hassio/backup.py:394-409` - Method: `on_job_progress()` - Processes event data and updates UI 5. **Blocking Wait** - Located in `homeassistant/components/hassio/backup.py:414` - Code: `await backup_complete.wait()` - Waits indefinitely for `backup_complete` event to be set - Event is set when `data.get("done") is True` in progress callback ### Where the Process Hangs **File:** `homeassistant/components/hassio/backup.py` **Method:** `SupervisorBackupReaderWriter._async_wait_for_backup()` **Line:** ~414 ```python async def _async_wait_for_backup(self, backup, locations, *, on_progress, ...) -> WrittenBackup: backup_complete = asyncio.Event() backup_id: str | None = None create_errors: list[dict[str, str]] = [] @callback def on_job_progress(data: Mapping[str, Any]) -> None: """Handle backup progress.""" nonlocal backup_id if not (stage := try_parse_enum(CreateBackupStage, data.get("stage"))): _LOGGER.debug("Unknown create stage: %s", data.get("stage")) else: on_progress(CreateBackupEvent(reason=None, stage=stage, state=CreateBackupState.IN_PROGRESS)) if data.get("done") is True: backup_id = data.get("reference") create_errors.extend(data.get("errors", [])) backup_complete.set() # <-- This never gets called unsub = self._async_listen_job_events(backup.job_id, on_job_progress) try: await self._get_job_state(backup.job_id, on_job_progress) await backup_complete.wait() # <-- Blocks here forever finally: unsub() # Code after this point never executes ``` ### What We ~~Know~~ _Believe_ 1. ✅ Backup job was successfully created in Supervisor 2. ✅ Home Assistant Core subscribed to job events 3. ✅ Initial job state was fetched and processed (showing addon_repositories stage) 4. ✅ Supervisor completed the backup successfully with no errors 5. ❌ Home Assistant Core never received events after the initial addon_repositories event 6. ❌ Home Assistant Core is blocked at `backup_complete.wait()` with no timeout 7. ❌ The `on_job_progress` callback with `done: True` was never called ### Potential Causes _(I'll spare you this as I can't confirm it to be reasonable)_ ## Impact ### User Impact - **High:** Users cannot create backups through Home Assistant UI - Backup appears to hang with no feedback - No error message or indication of what went wrong ### System Impact - **Medium:** Home Assistant remains in "creating backup" state - Backup manager is blocked and cannot process other backup operations - Resources held by waiting coroutine are not released - No timeout mechanism to recover automatically ### Workarounds Currently no known user-facing workarounds. The backup does complete in Supervisor, but: - HA Core is unaware of completion - Backup is not registered in HA Core's backup manager - Backup cannot be accessed through HA Core UI - HA Core backup state must be manually reset ## Root Cause Assessment _(More conjecture from here on down that I can't fairly evaluate removed.)_ ### What version of Home Assistant Core has the issue? 2025.10.3 ### What was the last working version of Home Assistant Core? _No response_ ### What type of installation are you running? Home Assistant OS ### Integration causing the issue Backup ### Link to integration documentation on our website _No response_ ### Diagnostics information [config_entry-backup-01K661PAHTFFV70GSA109ZJ7NN.json](https://github.com/user-attachments/files/23076861/config_entry-backup-01K661PAHTFFV70GSA109ZJ7NN.json) ### Example YAML snippet ```yaml ``` ### Anything in the logs that might be useful for us? ```txt ``` ### Additional information _No response_