Low-latency presence sensor: mmWave + PIR using ESPHome

This project has been the culmination of my focus as-of-late to deliver the fast responding, reliable motion lighting. Turning on and off lights is fun, except when there is lag, or a sensor misses movement. Building upon recent experience with ESPHome, I have refine the requirements to the following;

  • consistently rapid response time from detection to turn_on
  • sub 5 second turn_off time
  • the ability capture near-still movement
  • the ability to tune detection parameters
  • the ability to function in the absence of HA
    • reduced functionality accepted

Fulfilling all these requirements has been a journey, which has lead me to build this monstrosity. The requested maintenance window for a build picture was denied by ā€˜the bossā€™.

A few words on parts selection. This used to be a sole mmWave sensor used in conjuction with a Philips Hue PIR and that worked really well. 99% of the time. But the first requirement for fast response time was missed once or twice a day. Some device was caught sleeping and you could enter the room before the lights turned on. Leading me down the path of consolidating sensors and reducing dependencies.

I could never tell the true culprit of this infrequent slowness. Be it the ZHA network, wifi, HA or light controllers sleeping before being turned on. So I chose to remove the performance dependencies that I could, ZHA, HA & partially wifi by selecting a POE ESP32 for the sensor. So while the lights were still wifi, halving any potential wifi latency seemed like a good step.

Letā€™s talk about the sensors

* mmWave

The mmWave chosen was a known value for me, having used it for a few months in similar presence-based projects. The ability to capture micro-movement with sub-second hold times is a huge benefit. With the expected constraint of undesired detection due to any physical object moving. Hence the PIR combination.

* PIR

Adding a PIR sensor module was an unknown for me. And like most of my projects, I start cheap to see what works, what doesnā€™t, and what really matters. So when an HC-SR501 PIR landed at the doorstep, I quickly learned about electromagnetic interference and just poor detection qualities. After sorting out the EM issue by disabling wifi on the ESP32-POE, adding a choke and filter capacitor, the HC-SR501 was still an unreliable choice. The espresso machine never moves, but the heat from it would trigger a detection. This lead me to actually reading PIR datasheets and ohā€™boy are there a lot of options. I landed on a Panasonic Slight Motion u6A for a number of key reasons;

  • the 6uA seems to have the best balance of power draw and circuit [startup] stability time.
  • the slight motion speed detection threshold of 0.5M/s is desired based on experience from other radar projects and tracking people movement.
  • a smaller [target] object size helps trigger a detection sooner when appearing around a corner (head size vs body size target).

After two days of practical experience with the Panasonic I can confidently say that previous false positives (due to non-moving heat sources) are gone and the datasheets standby/mask times are conservative. I can see ~1 sec detection/redetection times which almost borders on flapping.

* Flapping & Detection

Since we are working with sensor modules with extremely quick trigger/hold/retrigger times it is quite important to include cooldown/debounce time into the light-trigger logic. For that reason I included the delay_off parameter in the binary_sensor configurations.

Letā€™s talk about the lights

The lights in question are running either Tasmota or MongooseOS (ShellyRGBW). I have considered migrating any and all to Tasmota or ESPHome but I am unconvinced the firmware is a culprit in any slow response. I say this from the observation that they all turn_on within milliseconds of each other. Therefore it stands to reason that any slowness in response is not firmware related. I have taken precautions non-the-less. These include;

  • disabling any power-save/ECO modes
  • configuring <500ms turn_on transition times
    • Tasmota speed 1 is the fastest available at 500MS
    • Shellyā€™s at 166ms because I can.

ESPHome Configuration

Letā€™s go back to our requirements and map out specific configurations targeted at solving each need;

consistently rapid response time from detection to turn_on

This requirement is met by having the sensor send http calls in order to turn_on the lights, even though HA has an automation to do so. No other parameters are sent to the lights from the sensor. The key here is that a light will begin to turn on sooner from a direct API call from the sensor than vs HA. HA retains all the customization options since there lights have numerous scenes/etc. Out of this I have gained at least a whole two tenths of a second. It could be more, but I only actually measured after the fact (no baseline was taken when sensor was wifi). If you watch Formula 1, you will know that that is a lot :slight_smile:

15:59:27.401 MQT: stat/StoveTasmota/RESULT = {"POWER":"ON"} <-- ESPHome
15:59:27.404 MQT: stat/StoveTasmota/POWER = ON
15:59:27.623 MQT: stat/StoveTasmota/RESULT = {"Fade":"ON"} <-- HA automation
15:59:27.638 MQT: stat/StoveTasmota/RESULT = {"Speed2":1}
15:59:27.651 MQT: stat/StoveTasmota/RESULT = {"POWER":"ON","Dimmer":100,"Color":"000000CC33","HSBColor":"0,87,0","White":100,"CT":222,"Channel":[0,0,0,80,20]}

sub 5 second turn_off time

This easily beats the Hue 8 second minimum as well as the Ecolink 5 second test mode. Plus the Ecolink is sloooow to trigger in comparison.

the ability capture near-still movement

This is achieve by a single mmWave sensor module. In the first build thread it was asked if two could be used in parallel. I was initially suspicious that they would interfere with each other as channel selection is not possible. Yet it seems they donā€™t. Assuming pointed in different directions as I have done.

The constraints

The first is obviously having to run a cable. But that is also true for the Aqara FP1 and it is yet to be determined its resilience to non-human micro-movements.

The second is heat. The -ISO version of the Olimex gets warm. I would be curious if the non-isolated version gets less so.

Whereā€™s the yaml!

esphome:
  name: poe-mmwave-pir
  includes:
    - uart_read_line_sensor.h
  on_boot:
    then: 
      # wifi interferes with HC-SR501 PIR generating false positives
      - lambda: WiFi.mode(WIFI_OFF); 
      
esp32:
  board: esp32-poe-iso
  framework:
    type: arduino

web_server:
  port: 80
  version: 2
  include_internal: true
  ota: false
  auth:
    username: admin
    password: !secret web_server_password
    
substitutions:
  device_name: poe-mmwave-pir
  stove: !secret stove_ip
  kitchen: !secret kitchen_ip
  kitchen2ch1: !secret kitchen2ch1_ip
  kitchen2ch2: !secret kitchen2ch2_ip

http_request:
  useragent: esphome/$device_name
  timeout: 2s

# Enable logging
logger:

# Enable Home Assistant API
api:
  reboot_timeout: 6h
  services:
      # Service to send a command directly to the display. Useful for testing
    - service: send_command
      variables:
        cmd: string
      then:
        - uart.write: !lambda 
            char buf[128];
            sprintf(buf, "%s\n", cmd.c_str());
            std::string s = buf;
            return std::vector<unsigned char>( s.begin(), s.end() );

mdns:
  disabled: true

ota:
  password: !secret another_one

ethernet:
  type: LAN8720
  mdc_pin: GPIO23
  mdio_pin: GPIO18
  clk_mode: GPIO17_OUT
  phy_addr: 0
  power_pin: GPIO12
  manual_ip:
    static_ip: !secret another_one
    gateway: !secret another_one
    subnet: 255.255.255.192
    dns1: !secret another_one
    dns2: !secret another_one

uart:
  id: uart_bus
  rx_pin: GPIO36
  tx_pin: GPIO4
  baud_rate: 115200

switch:
  - platform: template
    name: kitchen_lights
    id: kitchen_lights
    optimistic: true
    internal: true
    device_class: switch
    turn_on_action:
      then:
        - http_request.get:
            url: http://${stove}/cm?cmnd=POWER%20ON 
        - http_request.get:
            url: http://${kitchen}/color/0?turn=on
        - http_request.get:
            url: http://${kitchen2ch1}/white/0?turn=on
        - http_request.get:
            url: http://${kitchen2ch2}/white/1?turn=on 
    turn_off_action:
      then:    
        - http_request.get:
            url: http://${kitchen2ch1}/white/0?turn=off 
        - http_request.get:
            url: http://${kitchen2ch2}/white/1?turn=off 
        - http_request.get:
            url: http://${kitchen}/color/0?turn=off 
        - http_request.get:
            url: http://${stove}/cm?cmnd=POWER%20OFF  

binary_sensor:
  - platform: gpio
    name: mmwave_detection_lower
    id: mmwave_detection_lower
    internal: true
    pin:
      number: GPIO16
      mode: INPUT_PULLDOWN
    filters:
      - delayed_off: 2s      
      
  - platform: gpio
    name: mmwave_detection_upper
    id: mmwave_detection_upper
    internal: true
    pin:
      number: GPIO15
      mode: INPUT_PULLDOWN  
    filters:
      - delayed_off: 2s      

  - platform: gpio
    name: pir_detection
    id: pir_detection
    internal: true
    pin:
      number: GPIO14
      mode: INPUT_PULLDOWN
    filters:
      - delayed_off: 2s

  - platform: template
    name: mmwave_pir_combined
    id: mmwave_pir_combined
    device_class: motion
    filters:
      - delayed_off: 2s
    lambda: |-
      if ( ( id(mmwave_detection_lower).state or id(mmwave_detection_upper).state ) and id(pir_detection).state) {
        return true;
      } 
      else if (id(mmwave_detection_lower).state == 0 and id(mmwave_detection_upper).state == 0 and id(pir_detection).state == 0) {
        return false;
      } 
      else {
        return id(mmwave_pir_combined).state;
      }
    on_state:
      - if: 
          condition:
            - binary_sensor.is_on: mmwave_pir_combined
            - binary_sensor.is_off: motion_light_override
            - binary_sensor.is_off: kitchenswitch
          then:
            - switch.turn_on: kitchen_lights
      - if:
          condition:
            - binary_sensor.is_off: mmwave_pir_combined
          then:
            - switch.template.publish:
                id: kitchen_lights
                state: OFF 
      - if:
          condition:
            - not:
                api.connected
            - binary_sensor.is_off: mmwave_pir_combined
          then:
            - switch.turn_off: kitchen_lights  

  - platform: homeassistant
    id: kitchenswitch
    entity_id: switch.kitchenswitch
    
  - platform: homeassistant
    id: motion_light_override
    entity_id: input_boolean.motion_light_override
    
text_sensor:
  - platform: custom
    lambda: |-
      auto my_custom_sensor = new UartReadLineSensor(id(uart_bus));
      App.register_component(my_custom_sensor);
      return {my_custom_sensor};
    text_sensors:
      id: "uart_readline"
      internal: true

button:
  - platform: restart
    name: Restart $device_name   

# #Notes
# sensorStop
# # range almost 1m
# detRangeCfg -1 0 9 # kitchen lower 9
# # 5 sec disappear
# outputLatency -1 0 200
# #save
# saveCfg 0x45670123 0xCDEF89AB 0x956128C6 0xDF54AC89
# #start
# sensorStart    

[edit] - update lambda

My mmWave Projects

Tracking Radar
mmWave Presence Detection
Low-Latency DFRobot+PIR
mmWave Wars: one sensor (module) to rule them all

16 Likes

Great project! I was working on a presence sensor and testing this with a couple of different IR thermal sensors (MLX90640, AMG8833 and a D6T44L06) but couldnā€™t get it stable. Been thinking about mmWave also, your project is just the boost I needed to explore that. Iā€™m going to order parts and start with it asap.
Are you still using the DFRobot SEN0395 24Ghz mmWave sensor?
Thanks.

Glad to hear!

Thatā€™s the one. Have four deployed now, they have been working really well :slight_smile:

Great, thanks, Iā€™ll update as soon as I have one working

Do you have a link to buy ā€œPanasonic Slight Motion u6Aā€ ?

I purchased mine on Digikey. They are also available on Mouser.

There are other lens options available such as the wide angle version.

And three color options.

Search for ā€˜Panasonic EKMā€™ and they will show up :slight_smile:

I buy mine on Mouser (here). Their PIRs are excellent. Totally different league than the usual cheap HC-SR501 crap.

Hereā€™s a document about the different lens and sensitivity options they offer.

1 Like

Sorry to bother you again, at Mouser there are a lot of products on ā€˜Panasonic EKMā€™ , price ranges from $20 to $32. Can you share the part number that you are using?

I have tried to build a similar sensor combining a RCWL-0516 Micro Wave Radar Sensor and a HC-SR501 with limited success.

Our water heater sits close to where the sensor is and the sensor will trigger every time the heater does something.

I used the EKMB1309113K version. Highly recommended to review the document @HeyImAlex linked for all the additional options. Those other options include features such as movement across sensor zones. That may be more resilient to a large heat source than the single zone of the sensor I selected.

Likewise placement is very important as well. Movement along the X/Y axis is tracked far better than Y (approaching/receding).

This project has involved quite a bit of testing to ensure the best placement results. Especially with cats!

1 Like

Note that all the sensors in the document I linked are available in 3 sensitivities: standard, high and low. Thatā€™s not shown in the document, but you will find these variants on Mouser. The sensitivity is encoded in the first number of the product code:

EKMBx or EKMCx, where the x is one of these:

1 - standard sensitivity
4 - high sensitivity
7 - low sensitivity

Thereā€™s also a 2 for adjustable sensitivity, but thatā€™s only available with the analog versions which need additional electronics before you can connect them to an ESP.

1 Like

Having only been recently introduced to Panasonic PaPIRs with this project I was impressed by the array of options. And my experience using the first one impressed greatly.

There are other brands making good PIRs too, like Murata or TI. But the Panasonic ones are probably the best available right now. Itā€™s a little sad that people have come so accustomed and over-exposed to all these AM312s and HC-SR501s, that they think this is how PIRs are supposed to operate. Truth is, these cheap things that inundated the DIY space are unreliable low quality trash. Having constant false triggers is not normal for a PIR.

Iā€™ve been using the PaPIRs for a while now, I bought like a dozen variants with different lenses and sensitivities and they all work as expected for their use case. Even in harsh and difficult spots like outdoors. I still need to try the mmWave sensor you talked about, as the type of sensor fusion you did is absolutely the way to go for reducing unwanted triggers or making them more selective. Iā€™m just a little worried when you talked about the sensor being so sensitive it would literally trigger at anything moving. I guess Iā€™ll have to try ! Thanks for sharing your experience report, very interesting.

For some applications, this is perfect. Like a aimed at a bed for bed-occupancy. Or those midnight bathroom breaks when you donā€™t want the overhead light on while sitting frozen on the porcelain throne. For those low traffic scenarioā€™s, I have deployed the DFRobot solo with good success.

The kitchen on the other hand requires much finer control as there are heat sources, cats, fans, adjacent hallway through-traffic, etc that are undesired triggers for the DFRobot on its own.

I am glad that I stumbled upon the Panasonic PaPIR and that you have confirmed similar reliability.

I couldnā€™t agree more, I did some testing with an EKMB1304111K last couple of days and these PIRs are indeed impressive compared to others Iā€™ve tried in the past!

Glad they have worked out!

The black color model is pretty great too.

What modules have you tried in the past?

Quick note about EKMB vs EKMC selection. From plowing through Panasonic technical docs, thereā€™s a slight detection quality difference between both. The EKMB types are the low power consumption types (1ĀµA to 6ĀµA) mainly targeted at battery operated devices. The EKMC devices are for powered applications (170ĀµA). Since they have a larger power budget, they can do more advanced signal processing onboard and they donā€™t go to deep sleep mode. So theyā€™re slightly more reliable but they use more power.

Hereā€™s an interesting FAQ about good sensor placement, lens selection, outdoor proofing, etc.

I wonder if anyone tried the Panasonic Grid Eye sensors:

Panasonic Grid-EYEĀ® Infrared Array Sensors

Theyā€™re an inbetween thing between an advanced PIR and a thermal camera. They use a grid of 8x8 pyroelements (the PaPIRs use 2x2, normal low quality PIRs use just 1x1) and also measure absolute temperature like a thermal camera. So unlike a PIR, they can be used for static presence detection, they can detect multiple persons and provide information about movement direction. They sell for about 30ā‚¬ a piece and itā€™s an SMD package (not that easy to solder). But interesting device, maybe able to replace the mmWave sensor ?

1 Like

Those Grid-EYE look interesting. Too bad that the full module is twice the price of the mmWave module. Would be interested to hear the experience of others comparing them.

Iā€™ve never used them myself, but Grid-EYE are well-known by the room-assistant guys. They have an interesting wiki page about it if you want more info.

Yeah I also ordered a black one, but for use in bathroom/toilet I learned (from my thermopile/grideye experiments) that anything that looks like a camera will freak people out :laughing:
And most of us are used to white PIR sensors, so for those rooms I will use white of off-white PIRs.
I canā€™t really remember which other modules I used in the past, was many years ago, order some from amazon (SR501ā€™s I believe), did not like them, order some others, same problem, scavenged some old alarm PIRs I had lying around, nothing really worked well.
Grideye/AMG8833, MLX90640, and a D6T44L06 on the other hand - thermopile IR sensors - are a different thing, these do work. Had a AMG8833 proto board in one of our bathrooms for several months, almost 99% stable, but placement is key here, and the user facing part is, well ā€œlike a cameraā€.
So Iā€™ve been testing IR filter foil, lenses, etc anything that could make it look less like a camera, but that took a lot of time and did not improve the detection rate.
So that project went on hold for months, until I noticed your post about mmWave, and here we are. I really like the approach to combine initial detection with PIR.
I was thinking about combining mmWave and thermopile, but first letā€™s get the mmwave+PIR working :slightly_smiling_face:

Also check out TheRealWaldo, after initially trying to get the thermopile sensors working with some copy/pasting code from other sources using a webserver on an Arduino board I ended up doing most of my testing with his code and/or firmware for ESPā€™s GitHub - TheRealWaldo/thermal: Thermal Vision Sensor and Camera for Home Assistant