Disclosure: Supervisor security vulnerability

In my day job we have what we call a “postmortem”. We review things like “5 whys” of how the incident occured, work items to take on for future versions based on learnings, etc. What is great about this is 1) No blame, 2) Learning. Often, customers can request that we share our learnings in a document.

I would encourage the excellent core Home Assistant team to share postmortem learnings. From this, we users can learn as well.

Thanks for all of your hard work!

1 Like

Thinking just ”aloud”… I believe in SOCs that consider security, they have a trusted zone that contains all the security stuff ie stored keys and encryption/decryption. That stuff is not visible for other part of firmware. You could write some key to the zone, but cannot read it from there. Or, there is authetication for it. Functions that need keys, passwords etc, are located in the trusted zone.

So for example, when application/integration is installed, the credientials would be stored to the trusted zone and an identity would be allocated for them. When connection is needed, the integration would request it from a function located in trusted zone using the ID. Credientials would not be revealed but someone could still initiate connections to servers.

Thinking aloud ends, returning to reality. No idea could there even be a trusted zone for example in RPi.

1 Like

This disclosed vulnerability is obviously annoying but I do not consider it as being the only one in the HA codebase and others will appear in the future.

However the blame for working with cleartext passwords in the HA config file shouldn’t be put on HomeAssistant. It is an industry-wide issue, where countless cloud solution providers, although knowing it is not a good practice, still require users to authenticate with username/passwords. Access tokens or application-specific passwords are way easier to revoke and replace should any breach of credential data occur and allow external applications to work within a bounded scope.

Maybe to raise awareness among the HA user base would it be interesting to inform users of the potential risk to their credentials when they install components relying on static username/password.

Doing some analysis on my HA backup files I found a number of passwords stored in clear, most of them are not significant in terms of risk should they now be leaked, but I also found credentials for more important services and I’m glad that for those the two-factor authentication is enabled.

Applying a better-be-safe-than-sorry principle I will anyway change all passwords that were present in my HA configuration.

1 Like

This is exactly how I see it too. If it had been exploited we would have seen attempted attacks on things like connected email accounts etc.

I’d like to raise another question (bearing in mind my previous comment) before I get shouted down :wink:

Is it likely that this event might have raised the profile of HA amongst the bad actors and it will become a bigger focus of their efforts?

I only ask so that I am better informed. I have for a long time been considering an alternative external access method (e.g. VPN, Tailscale, CloudFare tunnelling(?)) but inertia caused by the absolute simplicity of NC has means I have made little progress in that direction.

(I suppose the first question should have been, would an alternative (to NC) external access method be inherently ‘safer’?)

That’s not an easy one to answer, as a lot of the answer comes down to what and how (and what threats you’re trying to defend against).

A correctly set up VPN solution (self hosted or Tailscale style) is likely to be the “best”.

A connection direct to HA using your chosen DNS provider is probably the “least”.

I doubt that Nabu Casa and Cloudflared bring significantly different levels of security. They’re both broadly similar in function, it’s just where the relay lives that’s the question. Who do you trust more?

A well set up proxy server of your own, where you’re doing filtering/other security, is potentially “better” than NC/Cloudflared - if you know what you’re doing. If you don’t it’ll be easy enough to make it “worse”.

I’ve never used NC for remote access, I’ve always handled it myself, with a proxy server that provides limited access to parts of HA (no remote UI, just enough for the app to work remotely, plus some webhooks for Tasker etc). I do (mostly) know what I’m doing, and am happy to spend the time dealing with when things don’t work. I’m not really typical :smiley:

IMO I’d summarise it as:

  1. Use Tailscale or your own VPN server if you’re really concerned - but this rules out integration with voice assistants unless you also set up a limited access proxy server
  2. Use Nabu Casa or Cloudflared if you want easy remote access without port forwarding - and are happy that it’s an all or nothing setup
  3. If you want direct access to HA from the Internet then use a proxy server, even if only for the logging - and consider limiting what you can reach remotely
3 Likes

It All depends upon how you use your system, which services you are running etc etc. I.E are Goggle connected to Ha, Services or Devices(Google_Home) , are any of you Devices connected to a another mobile app i.e TP-Link, netgear, D-link, Tuya, Asus(router-oops) etc. etc.

Ops, Tinkerer beats me :slight_smile:

2 Likes

I am going through the same thoughts on what actions to take

I too doubt the security flaw has been generally known in hacker communities because we would have this thread starting filling up with people experiencing stories of identity thelfs, ransomware attacks, etc

A foreign power could have planted malware to be activated later but I dount they would attack private Home Assistant installations. We are not interesting enough.

I went through my secrets file and found that 90% of sensitive secret content was actually no longer used either because yaml based integrations had turned into GUI and not using any secrets. I strongly suggest anyone to do the same.

Changing the HA password if your HA needs to be exposed to the internet is an obvious action

And then the real danger is where you have given Home Assistant access to the type of accounts where people can either steel your identity or hi-jack accounts that shut you out of your life. Or accounts where people can change a ship to address and start ordering stuff in your name

So services like

  • Google - Can you live with your Gmail email being taken over?
  • Amazon - besides Alexa that is a shopping thing
  • Cloud based integrations where the account being abused can mean stolen identity or purchases. A Philips Hue account is probably not going to be a huge disaster if attacked

I would be less worried about password to things that are inside your home like your MQTT server or your Door camera.

Some time ago I my Synology NAS hardware and also got DSM7
I chose btrfs as file system and enabled the snapshot feature.
I have a user account for admin only for DSM
And a user that can access files for everyday use.
And backups to external Hardware. And extra sensitive stuff like Family Photos and the music rips that took me 2 months are also backed up to additional hard drives that are normally not connected

If someone attacks us - they may do so while my Windows machine has a drive mapped to the NAS and they can encrypt all my files on the NAS. But this account cannot access snap shots or any admin tasks

In the event of a ransomware attack and encrypted NAS, I should be able to restore to a btrfs snapshot from before the attack.

And if the DSM has a vulnerability, I still have my off line backups. The end goal being that I can say f**k you to any ransomware attacker

I am putting my emphasis on anything that HA does on DSM like Samba shares or DSM integration. And I am addressing the credentials of the external service where I would not want to loose control of my account

But I am not going to panik about my ESP Home devices or my MQTT box.

It is a matter of personal choice of actions based on your personal risk assessment.

For my external access - I have removed my reverse proxy and no longer provide any external access. And I have disabled the HA remote control (the feature that I am still worried about being re-activated by an attacker gaining access to Nabucasa).

And all remote access is done with Wireguard.

Setting up Wireguard on your phones and tablets and setting it to only tunnel local 192.168 IPs works really well with the HA IOS app as well as the Android App.
You just setup the local IP address only and the HA app will use that and it gets tunneled through Wireguard. That is one single point of attach.

Many others use Cloudflare - I have no experience with that. And little trust as I do not understand it. I understand Wireguard and I trust the Linux kernel people that has implemented it so that is my choice of external access

There is no right or wrong here. I am just sharing my thoughts and actions

2 Likes

That logic only works under the assumption that potential attacks would have happened in mass recently or even after disclosure of the hole, AND that the affected persons would have seen this thread and answered in it.
This thing existed for years. Let’s for a moment assume that it was actually exploited a while ago. I don’t think that anyone who e.g. had a weirdly acting google account in 2021 would have linked it to HA. Or someone who lost access to their Google account altogether because of “suspicious behaviour” combined with a crappy recovery process on google’s side. Or a specifically targeted person with a security-critical job in a giant IT company who has all the reasons to sweep under the rug that the solution he uses at home to not have to get up from the couch to switch on the light indirectly was the entry gate to him losing crucial company secrets.

I’m not saying that any of that definitely happened. But in the same way you just cannot rule it out simply based on the logic that we would have heard more.

2 Likes

The thing is, that’s not the way state actors work. If they can mass infect the install base of an application easily, with very low chances of discovery, then they will do just that and silently leave a door open. Target or not, interesting or not. Just in case it may come in handy in the future.

I agree. Had I used the Supervisor in the past, I would definitely feel uneasy now. No immediate threat or anything like that. Just some nagging feeling I would not be able to get rid of. I would probably clean install the whole server, rotate all credentials, sanitize everything. Because you never know. 7 years is a long time.

1 Like

I am very paranoid by personal nature and work.

If I was not doing it before (which most of the following I am already doing), I would:

  1. review my Firewall rules for HA,
  2. remove port forward or mirroring for HA
  3. move HA to its own segment and prevent any off segment routing
  4. add VPN to edge device (router) explicit tunnel to HA-only segment,
  5. restrict all in- and outbound traffic to/from HA to just trusted devices
  6. Isolate the port on the switch
  7. get new SSD, and put clean HA on it,
  8. create backup of current running HA,
  9. scan backup with anti-malware tools and with eyes if the backups (textual stuff) contain anything unexpected,
  10. down the old HA
  11. restore config to new HA
  12. change everything in secrets.yaml
  13. make golden image of the new HA
  14. store golden image off network
  15. swap old HA with new HA
  16. download and reinstall all the pieces parts (e.g., integrations, HACS)
  17. fret what else could I have done.

Note that this is me, not you. Most people would not go this length to secure HA.

(I use the OS version.)

2 Likes

Right, but the point is that as far as we know, they can’t.
We as users don’t even have easy access to the underlying OS that Home Assistant runs on. The absolute best they could manage - would be to access the SSH terminal add-on and install their malware inside that, but it’s a docker container. And has no persistent storage. So changes that they make inside it, will not persist reboots.

As for suspicious Google account activity, I would have noticed, I have 2FA turned on, I would absolutely have noticed if I received a Google prompt that someone was trying to access my account, with a prompt to allow access. And while I might not have tied it to HA, I would absolutely remember it happening. It hasn’t happened.

3 Likes

You seriously should think a little “ahead” here, you should change credentials on all your device, integrations etc, as well as all on your phone, and everyone else in the household, also on all your email accounts, and change your “backup/restore-email” account, same goes for your Microsoft, google etc. accounts, might as well wipe your windows machine(s) to factory settings, androids also … just in case
If your not living on an isolated island, “best practice” would be to encrypt internal traffic as well

I will presume you are being funny :wink: so I will play along,

You seriously should think a little “ahead” here, you should change credentials on all your device, integrations etc,

Who says I do not? That said, service accounts & their secrets should be rotated automagically. On my to do list.

as well as all on your phone, and everyone else in the household, also on all your email accounts, and change your “backup/restore-email” account, same goes for your Microsoft, google etc. accounts,

I buy used phones, and reflash them. I do not have Google or Microsoft accounts.

might as well wipe your windows machine(s) to factory settings, androids also … just in case
If your not living on an isolated island, “best practice” would be to encrypt internal traffic as well

All my machines are secure boot, then 802.1x, and isolated on my local network. I have considered Opportunistic encryption (IPSec) but the overhead is too much. I am considering DMVPN instead.

I am hoping quantum locking of communications in peer to peer sessions but the hardware is a bit still pricey.

:clown_face:

There’s a lot more they could do. The Supervisor can, amongst others, update the Home Assistant container (and all others). So the first thing you could do is patch the local HA container. Installing a backdoor into the main HA process would be easy, and it’s internet facing so you have constant access to it. So this gives you full access to /config and to your entire local network. You can then patch the update service itself, so that future legitimate HA updates would remain backdoored. Next, you can harvest all credentials for locally connected devices and services. Anything IP routed is then potentially vulnerable, even if it’s cut off from the internet. IP cameras ? Use their own APIs to push a modified firmware onto them. ESPHome ? Push a contaminated FW update to your DIY ESP devices, infect ESPHome itself so the code generator will add malware into the binaries. It’s all open source, doing all this is easy peasy. A router, a firewall, a robovac, and AP, a smart TV, everything can then be scanned and potentially infected.

Basically, with the kind of access this bug would have given an attacker, they own your entire network infrastructure. The weak isolation provided by containerization won’t do anything about this.

2 Likes

Right, feel free to explain how though?
Literally everything, including the supervisor - is a docker container.
How do you propose that they can patch the docker container, to persist a backdoor in it, but it still allow it to pull down the official Home Assistant Core image - which would replace anything they did to the docker image? The ONLY way they could keep access would be by preventing Home Assistant from updating Core, which would be a massive red flag.

To be clear -
Home Assistant Core is a docker container
Supervisor is a container

As soon as Supervisor or Core is updated from the official registry, it will replace whatever local changes a hacker had made to the container, because it simply destroys the container and replaces it with the new image it has downloaded.

Furthermore, any major changes to the OS, would likely trigger that “System Not Healthy” issue, that we run in to continuously when we run the Supervised Install method, rather than the full Home Assistant OS.

2 Likes

Nice, i would love to see that mobile-phone or travel-laptop

Ultimately it depends on whether it is possible to use the Supervisor API to modify the Supervisor itself (I don’t know if that’s possible with the access this bug grants). Or if there is a way for an addon container to execute any code on the host OS with root priviledges, over a kernel call for example. Docker is containerization, not full virtualization. So if any of this is possible, it’s game over. The malware can then simply patch the official Docker images before installing them. But even that isn’t necessary to gain persistent access. They have access to your local network, they can install a backdoor into any other connected device, the access credentials are conveniently available.

Basically, as far as I see it, if someone gains access to your local network, consider yourself compromised.

1 Like

Maybe I missed it in the 238 replies. Has anyone from HA given a clear response as to what we should be doing other than updating to Supervisor 2023.03.1 or later?

1 Like

Yes.