Disclosure: Supervisor security vulnerability

Blog
347

I maintain my statement that this is a scandal.

They have been silent the days that went from publication of first fix till the actual fix came out.

They have remained silent about security ever since.

They have not reacted at all on the concern many of us have with the fact that a new security breach on the Nabu Casa side makes it possible for an attacker to enable remote access to their customers even though you have turned it off inside your Home Assistant.

It is hard not to think that they are hiding something. There is a reason behind the silence. My assumption is that there are more vulnerabilities which are not fixed. I am deeply concerned. And the worst thing for Nabu Casa is that it is only the paying customers that need to be worried about the possible remote access threat. Unsubscribe Nabu Casa and you are safer.

Also read carefully the PwnAssistant - Controlling /home's via a Home Assistant RCE

It is not only a couple of hysterical users in this thread. ELTTAM - security expers - criticize the communication to the users.

And to those only reading the latest posts. This little automation makes you at least a little bit safer in case you never use the NC remote access (link to earlier post)

2 Likes
348

I did, and it makes me feel a lot better…
because at the end they conclude this security vulnerabilty is correctly mitigated, even though the process itself could have been communicated better…

Also, they say if you want to be 100% sure, dont expose to the internet. Which is kind of moot, because that is what most users want/need. If you do, take extra precautions (they mention Tailscale, others have mentioned different types of services)

In my world, this comes nowhere near a Scandal, and as a fellow member here, I would really love to ask you to stop calling it that. It serves no purpose at all, and only distracts from our joint project-goal.

I do understand your point on the external access switch, and hope that could somehow be implemented. (personally I have a go-between solution for now: only open up when away from home)

Other than that, I hope the team will be more communicative next time, especially when requested. Although my main ask would be: prevent, and fix if necessary, with 100% focus on those.

your

borders the unacceptable to me, without any substantiation. If I were a dev, I would seriously think twice before returning here and throwing myself at the pack…

I’ve read somewhere else, and I would on the contrary prefer to have this trust in our HA/NC team:

If they could have done better at that point, they would have. Next time will be better still. There’s no way there’s not a next time in a project of this size.

at all times, until proven otherwise.

1 Like
349

The problem with this particular kind of vulnerability (RCE as root), this is not enough. If you think you might have been hit (and there’s no way to know), you must assume that any device connected to the network your HA box was on is compromised : your PC, your work notebook, your tablets, your IP cameras, your NVR, your NAS, your router and APs, even your DIY ESPHome type of devices. Devices with embedded firmware like IP cameras could be compromised in a way that makes it impossible to get rid of the malware through normal FW flashing procedures.

You would have to completely wipe the SD card / SSD using an external tool (don’t boot from it !) and then reinstall a clean image. This vulnerability gave root access to the host system. Consider anything on your SD/SSD tainted. You also can’t just restore a backup of your old config to the new system. You will have to manually audit all the config files (including the undocumented ones in .storage) one by one, line for line, for possible malicious changes. You cannot reuse custom components in the backup, you need to reinstall everything. In practice, this means throwing away all previous backups and restarting your entire config from scratch.

Remediating all the possible consequences of this vulnerability is a huge undertaking, if you think you may have been targeted.

2 Likes
350

so lets consider this a bit further. What would be the definition of ‘compromised’ here? Does that mean the devices software could be injected with malicious code? Or do you mean the possible malicious code in HA Supervisor could now connect to those devices (because of the credentials stored in HA)

If the latter, that would only require credential renewal?

What does interest me, and I havent read a lot about it in this community, is lan separation. What would be an ideal network config setup? should we allow all devices to see each other?

personally I have separate Lans for Cameras, IoT devices and some other grouped devices, and a ‘mother of networks’ that can see all. They are supposed to be separated networks.

Is there a guideline for an ideal HA setup on this matter? So, if an unfortunate event like this would ever occur again, we would at least have made it a bit more difficult to penetrate the full network.

Also, and that seems an under stressed configuration advice: dont use the default port 8123. Even the report linked above does not mention that as a first step-in making our HA system just that bit more difficult to penetrate.

Maybe these are questions beyond this topic. I do feel we could use some generic advice and link to these aspects if available elsewhere in the ‘Getting started’

351

Yes, that. The firmware code itself on the device may have been altered. A lot of malware specifically designed for this exists (remember Mirai ? There’s much worse now). The vulnerability gave a potential attacker the access needed to infect these internal devices.

Yeah that helps. But you have to keep in mind that the vuln gave an attacker full admimistrative rights over your HA system and all credentials you may have stored on it (like admin passwords to your vlan router). This can allow an attacker to easily penetrate the separation. You would have to build your lan as if it was an entirely untrusted network with internal firewalls etc. This can be a huge hassle.

Honestly, this is going to deter only the least skilled wannabes and some driveby bots. It won’t make your instance significantly harder to find for a moderately competent attacker. The safest way is to access your HA instance through a VPN.

1 Like
352

as Tinkerer explained long time ago. Which we have a community guide for on Securing Your Home Assistant Instance - #4 by silvrr

Must check this for clear enough instructions though, as even though these are well written up, I tend to get lost…

353

No, but it has cut the number of “drive by” attempts on my HA install to low single digits a year. That’s much easier to review in the proxy logs :wink:

2 Likes
354

I used to have a reverse proxy facing a subdomain to the internet which provided https for external HA (not on port 8123).
The minute this vulnerability hit I slammed the door.
I had meanwhile also added my own Wireguard VPN so it was easy to simply just VPN out phones and laptops to HA.
It is just that back door to Nabu Casa I cannot get fully rid of unless I unsubscribe or logout which means no Alexa bridge either.

The lack of trust comes from no communication from the leaders. I am not attacking the entire development community here. Paulus has chosen to keep silent on this issue. Last official Nabu Casa communication on this thread was Frenck on March 9th.

The real fix to the issue is a Supervisor release 2023.03.03 on March 27th. No communication at all.
So we do not know to what extent we were vulnerable in the period March 8 till March 27th.

And without any communication at all - also now - are we still vulnerable?

Trust is a human feeling and not a technical thing. Trust is built on human interaction. Silence is cancer for trust. This is not about what to do next time. It is about what to communicate now. We need an explanation of what additional exposure we have been subjected to from March 8th till March 27th. And we want to know what Nabu Casa is doing about the back door.

355

Unless I’m reading the blog posted by the security team that posted these vulnerabilities this sounds a little bit overblown. Some things that stuck out to me

Please correct me if I’m mistaken but it appears that if you are running HAOS (Home Assistant Operating System) then most of these vulnerabilities are a non issue and the biggest potential issues is specific integrations where the Integration provider handles the authentication and uses weak or poor authentication. I don’t believe this is something Nabu Casa/Home Assistant developers have any control over but please correct me if I am mistaken.

I have worked for a software company and if it connects to the internet, it probably has a security vulnerability. With that said, that’s why security teams and researches exist. They also must inform the software maker before publishing their results to give them time to patch the security vulnerability and it looks like it was handled in a very timely fashion based in the timeline below.

It appears to me that if you are running HAOS your okay outside possibly running integrations where the Integration provider handles the authentication and does it poorly. I apologize if I’m rehashing some things that have already been said but this is quite a long thread so it’ is difficult to go through every single post.

The below can be found HERE and list ways to run shell commands to find potential integrations that may pose a security risk.

Nothing is security proof and considering this was posted by a security research team who’s job is to FIND security vulnerabilities it’s hard to determine if these types of attacks were ever used in the wild. I would imagine most people here are running HAOS so based on the below, certain integrations COULD be an issue still but the Core and Supervisor security vulnerabilities have been patched and the security research team was no longer able to use the initial attacks after these were updated to the versions specified below.

Once again, if I’m mistaken in anything above please correct me. If you want 100 percent bullet proof security then cancel your internet because all software has some security vulnerability. Nothing is bullet proof and considering devices like this exist even not having internet doesn’t mean you are 100 percent secure.

Home Assistant can be installed in four different ways. These different installation types give users the ability to run Home Assistant according to their requirements and customise how much or how little is automatically managed.

The recommended installation method is via the Home Assistant Operating System (HAOS), which is a fully fledged Linux based operating system that runs the various Home Assistant components in Docker containers. This is intended to be run on devices like a Raspberry Pi, or within a virtual machine.
The standalone Home Assistant Container installation method is also recommended and provides a convenient way to run Home Assistant on a machine with Docker. This installation method does not come with the Supervisor component, so it misses out on a few features, namely add-ons.
The Home Assistant Supervised installation involves manually installing the Supervisor component on a Linux system which gives the full Home Assistant experience while letting the user manage the operating system themself.
Finally, the Home Assistant Core installation method is another manual installation in which the user runs the Home Assistant Core application directly with Python. As with the Home Assistant Container method, this does not come with the Supervisor component.
The three main components of a Home Assistant installation are the Home Assistant Core application, the Supervisor and the Operating System. All installations run at least the Core, while only the Supervised and Operating System installations run the Supervisor component. Since the Operating System component is only included in HAOS installations and because of its harder-to-reach attack surface, it was not an area of significant focus during our research. The other two components however, proved to be quite interesting to look at.

The three main components of a Home Assistant installation are the Home Assistant Core application, the Supervisor and the Operating System. All installations run at least the Core, while only the Supervised and Operating System installations run the Supervisor component. Since the Operating System component is only included in HAOS installations and because of its harder-to-reach attack surface, it was not an area of significant focus during our research. The other two components however, proved to be quite interesting to look at.

The Supervisor component is a Python program that lives in the home-assistant/supervisor repository. Its responsibility is to manage various parts of the Home Assistant installation by doing things like actually running/updating Home Assistant Core, managing backups, managing add-ons and even updating the operating system (when running a HAOS installation).

It exposes a HTTP API which is how the Core communicates with it. In the default HAOS installation, this service is not exposed on the network, so it is not possible to access this API remotely or even from within the same LAN.

As a simple example, the Telegram bot integration has a webhook endpoint which disables the default authentication check. It does perform some authentication in its own way though, in this case, based on the requester’s IP address:

The reason why this is interesting is because it means we have some fresh attack surface to look at. Integrations which perform authentication in their own way might do so poorly, which could lead to an authentication bypass…

A timeline of the disclosure process was as follows:

17/02/2023 - We begin researching the Home Assistant Supervisor Integration and discover the vulnerability
20/02/2023 - Vulnerability report sent to [email protected]
27/02/2023 - Follow up email sent to confirm receipt of report
28/02/2023 - Confirmation of receipt from Home Assistant
01/03/2023 - Home Assistant replies detailing plans to release hardening fixes, request a CVE and publish a blog post
01/03/2023 - CVE-2023-27482 reserved
01/03/2023 - Home Assistant 2023.3.0 is released, containing hardening in the HTTP integration security filters middleware
08/03/2023 - Home Assistant Supervisor 2023.03.1 is released, containing hardening in the security middleware
09/03/2023 - Home Assistant 2023.3.2 is released, containing further fixes in the Supervisor integration
09/03/2023 - Home Assistant publishes blog post and advisory
21/03/2023 - Bypass affecting Home Assistant Core <=2023.3.1 discovered and reported to vendor
21/03/2023 - Confirmation of receipt from Home Assistant
22/03/2023 - Home Assistant Supervisor 2023.03.2 is released, containing mitigation against the bypass
26/03/2023 - Bypass affecting Home Assistant Core <=2023.3.1 and Supervisor <=2023.03.2 discovered
27/03/2023 - Bypass reported to vendor
28/03/2023 - Confirmation of receipt from Home Assistant
29/03/2023 - Home Assistant Supervisor 2023.03.3 is released, containing mitigation against the bypass
03/05/2023 - Draft blog post shared with Home Assistant
04/05/2023 - Feedback on blog post received from Home Assistant
04/05/2023 - Advisory updated to reflect correct versions
10/05/2023 - Public release of elttam advisories and blog post

1 Like
356

To me it sounds like there was plenty of communication between the security research team and Nabu Casa/Home Assistant developers. Are you running HAOS in a VM or dedicated machine? If so it appears certain integrations which implement poor authentication are your only real worry. The security research team provides shell commands to find problematic integrations. See post above or read the full link below for all the details.

Since most Integrations are written by third parties this would be hard to go through every single one as there are now close to 2.5K supported integrations. Regarding your VPN question, the below was posted by the security team who found the vulnerabilities and it’s not known if any were ever used in the wild.

Security research teams must alert the software developer first to give them time to patch the issue before making it public, which they did patch in a reasonable time period. It’s also their job to find security vulnerabilities, most which have never been discovered before which is why software is constantly updated and patched when they are found yet most software companies don’t disclose this information either, especially if it’s patched in a reasonable time period which I believe is typically 3 months before the research team can go public with their full findings.

Any software that touches the internet has security vulnerabilities and you would be reading articles all day if you went through every security patch for every single piece of software you run. To me it looks like this was handled very well and within a very reasonable time period between the HA developers and the security research team.

We encourage users to consider VPN services like Tailscale which make remote access simple and secure.

Since the Operating System component is only included in HAOS installations and because of its harder-to-reach attack surface, it was not an area of significant focus during our research. The other two components however, proved to be quite interesting to look at.

357

You are mistaken :slightly_smiling_face: The vulnerability is/was in the supervisor integration. This integration, which is natively part of HAOS and Supervised installs connecting the Supervisor container with HA as a kind of glue layer , allowed the unauthenticated root access to your system. In fact, running HAOS exposes you the most to this vulnerability. Supervised too. Container and Core installs were not affected.

The part about integrations other than the supervisor one handling their own auth was another potential attack surface the researchers outlined. They did not explore it further due to time constraints. They went for the low hanging fruit in thr supervisor integration.

1 Like
359

Then why was the below posted in the security research team who found the vulnerabilities? Not trying to argue, just genuinely curious as they specifically said HAOS wasn’t affected by this (outside integrations) If the 1HTTP API isn’t exposed on the LAN or over the internet when running HAOS then why would it still be a security vulnerability?

The three main components of a Home Assistant installation are the Home Assistant Core application, the Supervisor and the Operating System. All installations run at least the Core, while only the Supervised and Operating System installations run the Supervisor component. Since the Operating System component is only included in HAOS installations and because of its harder-to-reach attack surface, it was not an area of significant focus during our research*

The other two components however, proved to be quite interesting to look at.The Supervisor component is a Python program that lives in the home-assistant/supervisor repository. Its responsibility is to manage various parts of the Home Assistant installation by doing things like actually running/updating Home Assistant Core, managing backups, managing add-ons and even updating the operating system (when running a HAOS installation).

It exposes a HTTP API which is how the Core communicates with it. In the default HAOS installation, this service is not exposed on the network, so it is not possible to access this API remotely or even from within the same LAN

360

  1. They did not research attack vectors on the underlying OS (Buildroot) itself directly. They focussed on attack vectors through HA (which can still reach the OS though).

  2. The Supervisor is made of two parts. The Supervisor container for one, this is what they are talking about in this passage. It exposes a http control API that is not normally exposed to the outside world (keep the italic part in mind for later). The second part is a Supervisor integration which connects the Supervisor container to your HA instance. It uses the internal Supervisor http API to control it, get status, etc. It’s a bug in this supervisor integration that accidentally exposes the internal supervisor http API to the outside world, without authentication.

1 Like
361

You are technically correct (which is the best type of correct) although the security team who wrote the blog post above could of made it more clear as this was in the middle of the blog

It exposes a HTTP API which is how the Core communicates with it. In the default HAOS installation, this service is not exposed on the network, so it is not possible to access this API remotely or even from within the same LAN.

But then this was the last paragraph in the blog after all the integration stuff (and Supervisor is an Integration) I think they could have added those closer together or added something to the above because they almost made it sound like the HAOS install wasn’t affected but it is The only thing in the blog after the below was their timeline of communication between HA developers. On first read I thought that meant if you were running Supervisor on a non HAOS install but I was obviously wrong in that assumption. Thanks for clarifying.

Ultimately, the vulnerability is exploitable as long as the Home Assistant instance runs the Supervisor component with the Supervisor integration and is reachable via the internet or through local network access – random hostnames or TLS do not provide any form of protection in this case. The simplest way to avoid being at risk from a remote attacker is to not expose the instance to the internet. We encourage users to consider VPN services like Tailscale which make remote access simple and secure.

362

Yeah it could have been worded a bit better. But then again, these reports are pretty technical by their very nature.

I think it would have been HA/NC’s job to summarize all this in simpler terms and post it in the advisory (regardless of how bad the news would have been), with simple and easy to understand bullet points of who and what and when and how things were affected.

3 Likes
363

Hi guys,
It’s part of my day to day job to deal with vulnerability management and what I can see that the vulnerabilities reported were dealt by HA team very well.

70% of this community are using Apple iOS devices, have you seen how Apple is reporting and communicating about critical vulnerabilities affecting their products?

The fact that HA/NC did not publish a final statement is a very normal practice as usually the third party researcher will publish the finding.

This vulnerability should have been categorised as a Zero-Day with no Public Exploits as there is no evidence of someone abusing this vulnerability.

A little real example.
Was dealing with a software developer for a banking web application and after pen-testing it was identified that by manipulating the URL you could jump to someone elses session and see and manipulate the data and when I have made a statement that the software has vulnerabilities I was reported to people on higher food chain that I was rude to the software developer.

The moral is the all the software has vulnerabilities and we should not be rude to developers, have you tried to write software yourself and make it open source.

Also have you seen the latest May Microsoft Patch Tuesday vulnerabilities?

Please stop being rude to the Software developers at HA/NC

5 Likes
364

VPN into your network or nothing for something like home assistant and lock it all down as much as possible. it’s just too critical/sensitive. i chortled when i saw cloud/nabucasa the first time. kind of a catch 22 though as most people using those solutions probably don’t understand all the implications of trusting those services or what a VPN is. they just know their wife won’t bitch as much if she can use it away from home easily.

365

Hmm I reverse proxy through cloudflare then again through my synology… I think I’m pretty secure!

IMG_1223

1 Like
366

This is a terrible argument.
I can’t build an entire car from scratch myself either, but I still expect the brakes to work correctly on the one someone else built for me, and I have the right to be “unhappy” about it if they don’t.

2 Likes
367

Nothing is perfect… especially software!! … IMG_1224