HAOS bricked with 14.1 update (proxmox, "operation=mkdir, permission denied")

Installation
1

As with the other posts (e.g. here, here, here, and here) – apparently the HAOS 14.1 upgrade bricked my Proxmox VM – yet it seems to be a different error, that’s why I’m writing another post.

So, I get tons of this in the logs (journalctl in the “emergency prompt”):

# both are originally one line – broken up for readability
AVC apparmor="denied" operation="mkdir" class ="file" profile="hassio-supervisor" 
name="/usr/local/lib/python3.12/site-packages/jinja2/__pycache__"
comm="python3" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
# next line
SYSCALL arch=c000003e syscall=83 success=no exit=-13
a0=7f66e3c92e70 comm=python3 exe="/usr/local/bin/python3.12"
subj=hassio-supervisor key=(null)
# ... and now repeat those two a LOT of times ...

Some notes:

  • the Proxmox VM was created using “proxmox-helper-scripts” not two months ago
  • I tried updating the OS before Home Assistant by hitting “Update” in the iOS app’s UI
  • when restarting, I get two “slots” in the boot manager – both fail to start like this:
    Waiting for the Home Assistant CLI to be ready...

[WARN] Home Assistant CLI not starting! Jump into emergency console...
  • the ha command does not work in the shell:
    # ha
Error response from daemon: No such container: hassio_cli

Can anyone help? maybe I just have to create that dir … but … which one?!

(also – lesson learned: remember to never hit “upgrade” in out of a whim again without a VM backup …)

2

Have you tried physically power-cycling the hardware that Proxmox is running on? I’ve had several odd issues after HA updates that I could only resolve by rebooting the Intel NUC where Proxmox is installed.

3

No problems with Proxmox and HAOS 14.1 here. So, it’s not a universal problem.

4

will do now, if you don’t read anything else in 20 minutes time it was not a success :wink: .

it must be nice to simply define yourself as “universal”. :roll_eyes:

5

Just trying to say not everyone running Proxmox will have problems with 14.1.

6

I mean, do you really think this was an even remotely helpful comment in any imaginable unifverse? :man_shrugging:

Anyway, I killed my installation, and I’m re-doing it. Sucks a lot, but it’s faster than trying to debug the shit.

Also: Backups from now on.

7

Yes, I think that comment was helpful. Because it shows that this problem is something specific to your installation (and possibly other people’s too of course). It’s not a given that 14.1 will break Proxmox installations no matter what. So the comment could guide you to look for the specifics of your installation that together with the changes between 14.1 and 14.0 (I think that’s only the Linux kernel?) caused the problem.
I’ve had a similar situation with Virtualbox and a HAOS update. Turned out that there were changes in the Linux kernel that specifically caused problems with VirtualBox 6, which was what got packaged with Ubuntu at that time. It wasn’t anything HA specific, there were totally different guest OS’s affected too. I found out about it through the Virtualbox forum, not via HA.

1 Like
8

Since you are using Proxmox, you should be creating a Snapshot before doing any updates.
That gives you a simple roll-back process if things don’t work out.

9

agree to disagree then :man_shrugging: . don’t know how that would have helped, and i’m fairly sure the HA people tests the stuff before it goes out – so I was not assuming a “fundamental” problem here.

also if you read my post, it apparently was a “permission denied” error, apparently, on the shell level – which has rarely to do with any kernel & virtualization issues.

aaanyways, it was a very late night, and it’s almost fully restored, so this is now officially no longer a thing.

10

yah, no shit :joy: . but once you hit backup every so often without a glitch … y’know … :wink:

11

This ^
I trust fixing something gone wrong by restoring a Proxmox snapshot much more than trying to revert to a HA backup. 10 times faster as well!