Wi-Fi sensing in Home Assistant

Social
1

There have been a ton of articles these past months about how Wi-Fi can be used to identify you, track you, see through walls, do pose detection, breathing detection, presence detection, etc. As someone working with Wi-Fi sensing, I get a ton of messages from people asking me if certain projects are scams and whether many of these grand claims are true. So I thought I would make a post about the current state of Wi-Fi sensing, what is actually feasible today, what is mostly working in lab conditions and not production, and what to be aware of to avoid spending a lot of time setting up projects that end up not working.

I’m the founder of a Wi-Fi sensing system. It is closed source, proprietary and paid, which rubs some people the wrong way, so I’m not going to use any names or links in this article to avoid it being about that. I’ll just try to provide some guidelines as to what to look for when researching solutions if you’d like to start using Wi-Fi sensing in your setup and what should raise some red flags.

On a side note, I’ve been working on this technology for the past 4 years, and I can’t help but notice how the number of articles being pushed on Reddit, Hacker News, etc. about Wi-Fi sensing these past 6 months has exploded. The new Wi-Fi standard for enabling the infrastructure of Wi-Fi sensing (IEEE 802.11bf) was recently formalized, so that explains part of it, but I suspect it comes down to the fact that a lot of routers already have or will start shipping some of these features soon and there might be an awareness/marketing campaign going on, although I don’t have any proof of that.

Most of these grand claims about features of Wi-Fi sensing are on some level true. It can be done with the exact right hardware (number of antennas, bandwidth, devices talking together in the right way), in the exact environment it was trained on. It’s therefore possible to have some very wild results in lab conditions that make for great article headlines but are nowhere near production ready on the current hardware in your typical home.

The production-ready solutions today are much simpler and will mostly come down to motion and presence detection. Heart rate estimation, sleep monitoring and fall detection are possible, but there aren’t any working solutions for Home Assistant as of now.

Wi-Fi sensing systems today are either based on:

  • ESP32s as they allow extracting the Channel State Information needed
  • Firmware in the router. This is already shipped in some routers and will come to many in the near future. But it’s heavily patented and might not be implemented in, for example, OpenWrt for a long time.

Most routers today that include Wi-Fi sensing only detect motion. I haven’t seen any offer more advanced features, but it’s probably coming. The router has the benefit of multiple antennas and high bandwidth, which opens a lot of possibilities, at the cost of loss of privacy. The advertisement of features in a router comes down to your trust in that brand, but if a router from a reputable brand is released which claims advanced features I wouldn’t be hesitant to believe them. But for now you are mostly going to see motion and presence detection.

ESP32 based projects are the ones to be skeptical about. I’m not aware of any working project that advertises anything beyond motion and presence detection. ESP32s on the market today have only a single antenna and very low bandwidth, so precise localization, people counting, etc. are not really possible. It’s possible to implement breath rate estimation, fall detection and sleep monitoring using ESP32s, but I’m not aware of any working project that does so yet.

With all of this said, Wi-Fi sensing is going to be a lot more mainstream in the years to come with many use cases for Home Assistant. But for now be aware of grand claims and if you can’t find a reputable review, video on YouTube, etc., then it’s probably not a working project.

8 Likes
2

I have not come across the term “Wi-Fi sensing”, but have seen mmWave 60GHz and 24GHz sensors … and wondering if your comments are including them ?

I assume they are both based on the same basic physics (electromagnetic waves) - but wi-fi is implemented as a broadcast communication; whereas my understanding of the mmWave sensors is that it is implemented like radar measuring the time that a signal bounces back. Are you seeing much more crossover between these approaches ?

3

On the mmWave half of that question, since it is still open.

The 24 and 60 GHz modules most of us use are FMCW radar. They sweep a frequency ramp and measure the beat frequency of the reflection, which gives range, and the phase shift across frames gives velocity. So it is not quite time-of-flight: the timing is inferred from frequency rather than measured directly.

The practical part that surprised me is how easily they go through interior walls. In my living room, 1.5% of four days of target coordinates came from the room behind the nearest wall, and no sensitivity setting removed them; only knowing where the wall was did. That is the same property that makes the through-wall Wi-Fi claims plausible, and also the reason “presence in this room” is harder than it looks with either technology.

I have not worked with Wi-Fi CSI, so I cannot say how much of the processing carries over.

4

One concrete data point for the “sees through walls” claims, since that’s the part people get excited about. My 24 GHz radar kept showing a second person past the living room wall and I assumed it was through-wall detection. It wasn’t: after plotting 275k coordinate pairs, the extra targets sat exactly where mirror images of the real people would be, three real/phantom pairs agreeing on a reflecting plane within 85 mm of each other. So on a dedicated radar with a clean range estimate, the most convincing “through-wall” signal in my house was a reflection. I’d be curious how a Wi-Fi CSI system separates a reflection from a body, given it has a lot less to work with than a chirp.

5

I am no expert when it comes to mmWave, but @sglim’s explanation also matches my understanding of how it works.

Wi-Fi sensing isn’t using dedicated sensing devices like mmWave which sweeps a frequency range. Instead it takes advantage of the fact that modern Wi-Fi uses OFDM, which in simple terms requires the receiving radio to “tune in” to some known pilot tones on every received packet. The amount of tuning required to see these pilot tones is highly dependent on the environment, and even subtle human changes (we are mostly water) change how much tuning is needed.

Wi-Fi uses a much lower frequency band (2.4GHz, 5GHz) than mmWave, which means it penetrates walls easily. Whole-home presence detection is possible with a few sensors compared to mmWave which needs to “look” at the area it is sensing in.

Wi-Fi sensing also has the benefit of working on existing hardware e.g. ESPHome devices or by analyzing traffic from smart devices in the home (Computer, TV, Playstation etc.).

6

That matches my one hands-on run with the RSSI end of it, WiZ SpaceSense in a New York apartment a few years back: two bulbs per room watching the signal between them, and it caught me more reliably than I expected. In fairness the place was a studio with one bathroom, so “is anyone in this room” had exactly two possible answers and I can’t claim I stress tested it. I’ve also done the other “no dedicated sensor” trick, fingerprinting which appliance just switched on from the whole-home power draw, and it has the same shape: you get a clean delta with no address on it, and “what” only comes after you’ve built a library of labeled signatures. So my guess is Wi-Fi sensing wins on coverage and loses on confidence, since a tuning delta has no range or angle to sanity check against, while the radar at least hands you a coordinate you can prove wrong. One practical question, since you mentioned existing ESPHome devices: does CSI capture coexist with an ESP32 that’s already running as a Bluetooth proxy, or does it want the radio to itself? All four of mine are doing BLE for Bermuda, so that decides whether I can try this without buying boards.

7

You are right that it wins on coverage against mmWave and loses on localization. I wrote a mmWave vs Wi-Fi sensing article a while back which goes into more depth: mmWave vs. Wi-Fi Sensing

Regarding ESPHome devices, yes bluetooth proxy can work alongside it. But since the BLE stack consumes a lot of RAM, you might need to enable PSRAM depending on the board to avoid out-of-memory issues.

One thing I’m working on right now is support for passive devices, meaning one dedicated ESP32 with the firmware on it and then analyzing traffic from devices without the firmware (a smart device, playstation etc.). That should reduce the number of boards needed for each zone.

8

Thanks, that settles it for me: three of my four are C3s, which don’t have PSRAM at all, and the fourth is a plain WROOM that’s already tight on RAM just running the BLE proxy under ESP-IDF. The passive mode is the part I’d wait for. Half the boards in this apartment are bolted in place doing other jobs, so “one dedicated board and listen to everything else” fits how homes actually end up wired.

9

I am curious about this topic because I just saw a video of some tests with esp32 boards.

I was wondering if it would be possible to make a simple presence detection as an alarm system.
Our homes now have dozens or wifi devices, usually with a few esp or esp32 devices.
Add multiple APs and it should be possible to calculate obstacles between the devices and the APs, right?

It would probably need a base line, like initial recording of wifi behavior when no one is home. Since most IoT devices are stationary, this would probably allow mapping of the apartment.

Just wondering if a “simple” project like this, essentially a large scale presence detection would be doable without special equipment or newest hardware/wifi components.

Heart rate and stuff like that, well, i have my doubts that those would work reliably even with special equipment (on a large area like an apartment or house). But mere movement of large objects like humans… might be interesting.

Why would processing be needed on esp devices? Could the RPi or miniPC running home assistant not do all the calulations and even run a small AI agent for this task? The esp devices would only need to broadcast their wifi info and the APs running e.g. OpenWRT would supply their info on the clients. Rest would be handled by the brain, i.e. miniPC/RPi.

10

That is very doable, and the working Wi-Fi sensing systems today does that already. The reason you need an ESP32 is that extraction of Channel State Information (CSI) is necessary to feed the algorithms that detect presence, and ESP32 happens to provide that through it’s driver and is cheap to acquire. The router is also capable of CSI extraction, but it’s access is very limited to a few models through custom firmware.

11

Would a mixture of CSI from ESP32 devices combined with RSSI from ESP8266 devices work? That way you could use all devices, from old to new. A lot of people still have Shelly devices and other old ESP devices installed all over the house.

12

The solutions I know of only works with ESP32s flashed on all devices. I’m working on using passive devices with some promising results. A beta version of that might be available within the next month or so. That will allow using other Wi-Fi devices such as old ESP8266, computers, smart TVs etc. You would still need at least one ESP32 flashed though.

13

I think OpenWRT routers could supply CSI instead of esp32 devices. Probably with some advantages that APs bring (and much more compute power).

Do you or does anybod know of any ongoing open source projects I could check out?

14

It might be possible with some chipsets, but as far as I knows it’s not a feature of OpenWrt. I have listed the projects I know in this article. There is also an open source one you could try.

15

WiFi sensing monitors the data packet amplitude and phase shift from packet to packet. Together it makes a CSI, Channel State Information. If you walk into a room your body reflects the RF from the AP differently which is seen as a phase shift, thus a change in the CSI.

The IEEE 802.11bf standard formalizes Wi‑Fi sensing in routers. It is a published standard but not yet implemented.

I have read of people doing WiFi sensing using the ESP32. I thought of experimenting with it just to sense if no one is home. If the CSI doesn’t change much (noise in the data) then no one is home. If someone walks into (or near) the home, presence is detected. I initially thought of using it to throttle the AC if no one is home (like the Nest thermostats). But someone walking on the sidewalk, a car on the road would also alter the CSI.

16

If you want to experiment with creating your own solution then esp-csi is the official example repo from Espressif.

17

Atheros and mediatek chips seem to be suitable (at least some). Those are also the primary chips supported by OpenWRT.

I’ll wait and see if I find some time later this year to look into it. Or maybe someone smarter will have started a project by then.

18

No CSI here, so I can’t answer the mixing part directly. What I can tell you is what plain RSSI costs you once you ask it for rooms: over the last 111 hours my BLE room sensor, fed by four ESP32 proxies, changed rooms 16.3 times an hour, which is a room change every 3.7 minutes around the clock. Reading it once every 30 seconds brings that down to 6.9, and a layer that only judges when at least three of the four proxies actually report a distance brings it to 2.1, though two thirds of the time that layer is holding the previous room rather than deciding anything. So with RSSI the filtering is most of the work, and I’d expect old ESP8266s to need the same treatment.