mmWave Target Tracking - fully customized ESPHome support!

TL;DR - I took a tracking radar sensor + an ESP32 and made an individually addressable light strip follow me as I walk by. am making a Nerf CWIS for my desk! (it’s a work in progress…)

Tracking servo’s deployed!

Introduction

This project is a confession of what happens when ambition exceeds skills. Having successfully deployed ESPHome projects based on DFRobot and (other) RFBeam mmWave sensors for presence detection, it was time to tackle real-time radar targeting!

Having been slightly disappointed with the RFBeam K-LD2 that provides GPIO-based detection of approach/receeding/left/right target parameters, it was time to upgrade to the RFBeam K-LD7 radar sensor module that provides target distance, angle and speed!

This is what they sell it for; how boring!
image

With excited goals of creating a Nerf CIWS to keep those pesky furry felines off my work desk, I dove straight into a ESPHome brick wall.

my hopes - an HA controlled sentry…

reality - what do I do with this again…!?


…yes I really did Dremel off the wifi antenna…

K-LD7 ESPHome Support

ESPHome has very wide support for sensor modules and when there is no native support, custom UART device fills in the gaps. Having been successful in all projects to date with no C++ skills, how hard could it be to copy & paste my way to success?

After months of poking away at it, the HA Community came to the rescue. Before we got there, it had been a struggle.

Python Support

As it turns out, there is excellent Python support for the K-LD7 which introduced me to using bytearrays and hexadecimal data. That library can be stripped down and works very well in CircuitPython on an ESP32; even at very high baud rates over wifi.

A New Hope

Having shelved the ESPHome side of the project for a while, help soon came from @phillip1 in developing the C++ .h necessary for reading and parsing, not only target data, but configuration parameters as well.

How does it get targeting data?

A vastly oversimplified explanation is;

  1. an analogue modulating mmWave transmitter emits a radiation pattern

  2. two I/Q receivers sample ADC data

  3. the onboard MCU performs FFT magic…

  4. candidate targets above threshold are generated

  5. a single track is generated for speed, angle and distance for the most dominant target

Sensor Placement

It is important to understand the relation between sensor orientation and the angle output. The K-LD7 uses a 2D array which means that depending on placement, angle is either degrees “left and right”, or “up and down”. But not both. Keep your eye open for 3D tracking mmWave sensor modules (this year?).

Capturing Target Data

All this magic is performed locally on the K-LD7 with a long list of parameters that can be used for target discrimination. The datasheet goes into detail of all hex values that the sensor accepts and reports. More on this later.

The challenge is getting them into ESPHome at a default eight times a second (~125ms per target reporting interval). When you have an unsupported sensor using a custom device, custom parsing code must be written in C++.

Target Speed to Refresh Rate Relationship

There is an inversely proportional relationship between target speed resolution and sensor refresh rate. Depending on the intended installation, this is an important parameter to consider as it includes a power-consumption impact.
image

ESPHome Custom UART Device

From the YAML perspective, it seems so innocent. Yet there is lot that goes into making these four sensors available. Please note that since that target data can potentially update up to eight times a second, they are marked internal: true in order to preserve the default History configuration of Home Assistant. This is easily changed if the use-case warrants. In the example provided, HA does not need the target data and is only used for configuration and OTA.

sensor:
  - platform: custom
    lambda: |-
      auto s = new kld7(id(uart_bus0));
      App.register_component(s);
      return {s->distance_sensor, s->speed_sensor, s->angle_sensor, s->db_sensor};
      
    sensors:
    - name: distance_sensor
      id: distance_sensor
      internal: true
      unit_of_measurement: cm
    - name: speed_sensor
      id: speed_sensor
      internal: true
      unit_of_measurement: kph
    - name: angle_sensor
      id: angle_sensor
      internal: true
      unit_of_measurement: deg
    - name: db_sensor
      id: db_sensor
      internal: true
      unit_of_measurement: dB

Radar Parameters

But wait, there’s more UART data we can use! The radar is highly configurable. So much so that you can get yourself into situations where nothing can possibly be detected! Study the datasheet parameter chart carefully for best results.

In order to tune these parameters, each setting is configurable in HA and the sensor even synchronizes these settings on every boot. There is also a factory_reset option in order to start over should you get into a situation where zero targets are detected due to self-cancelling parameter settings.

What do we do with it?

Anything you want! As a example, I have included the code that takes the targeting data and uses WLED UDP Realtime Control to light up an LED based on distance data. This very simple example is but one of many possibilities that are made available when you can track an object in 2D space.

Custom Device C++

A thank you goes out to @phillip1 for the support as I demonstrated all the traits of a C++ noob. I have learned a lot and wanted to highlight some of code. While the ESPHome documentation is great to get you started, knowing your types, conversions and general programming principles helps a lot.

Sensor Application Response

Every sensor command issued by a yaml uart.write returns a dedicated hexadecimal response. Validating a few specific response is performed by the following;

    bool doesHeaderMatch(std::vector<uint8_t> bytes, std::string header, int payloadLength) {
        if (bytes[0] != header[0] || bytes[1] != header[1] || bytes[2] != header[2] || bytes[3] != header[3]) {
            return false;
        }
        // ESP_LOGD("custom", "header matched: ", header);
        return bytes[4] == payloadLength && bytes[5] == 0 && bytes[6] == 0 && bytes[7] == 0;
    }

Should you wish to add responses in the datasheet that we have yet to, a matching struct/union is needed with the appropriate bytes sizes from the datasheet.

    typedef struct {
        uint16_t distance;
        int16_t speed;
        int16_t angle;
        uint16_t db;
    } TDAT;

    typedef union {
        TDAT tdat;
        uint8_t bytes[sizeof(TDAT)];
    } TDATUnion;

Hex Parsing and Publishing

The combination struct/union make publishing of the data quite concise;

            TDATUnion tdatUnion;
            std::copy(bytes.begin() + 8, bytes.end(), tdatUnion.bytes);
            // ESP_LOGD("custom", "Sending TDAT");
            distance_sensor->publish_state(tdatUnion.tdat.distance);
            speed_sensor->publish_state(tdatUnion.tdat.speed / 100);
            angle_sensor->publish_state(tdatUnion.tdat.angle / 100);
            db_sensor->publish_state(tdatUnion.tdat.db / 100);

Smart Servos

This needs a post on its own - but the Lynxmotion LSS work very well w/ ESPHome and save you from having to write your own PWM PID / rotary encoder things…

What’s left?

Semi-auto Nerf is next…

Todo

  • a few of the configuration parameters use non-linear index-to-result parameters. ESPHome select could make this more human readable but isn’t quite baked yet as it.

The Code

Moved to first reply.

Appendix

Caveats and Lessons Learned

  • adding too much debugging will slow down target tracking publishing
  • ESPHome UART debug makes the sensor read unreliable

FAQ

  • How do I wire the sensor?
    ** Pinouts are in the datasheet
  • I used your pins and it didn’t work
    ** yup :wink:
  • Wifi doesn’t work!
    ** replace my Ethernet section with an appropriate Wifi section in the YAML
  • How does this C++ deployment compare to the CircuitPython one?
    ** they both work very well, ESPHome is just easier to OTA and control w/ HA
  • Do I need an Ethernet ESP32
    ** No
  • What about this_other_random_mmWave_sensor?
    ** buy it and tell us!
  • Where is the turret build?
    ** :wink:
  • Were is the WLED usermod for the K-LD7?
    ** please go submit one!

More Project Ideas

  • calculate relative coordinates using distance and angle to plot robot vacuum movement
  • creepy doll/picture with servo controlled eyes that follows you as you go by?
  • outdoor servo-controlled spotlight that lights up and tracks (perhaps 3D is better for this)
  • a Lovelace card that plots a live track at x interval…

My mmWave Projects

Tracking Radar
mmWave Presence Detection
Low-Latency DFRobot+PIR
mmWave Wars: one sensor (module) to rule them all

16 Likes

[EDIT LOG]
6-7-2022:

  • add configurable WLED RTN (return to normal) setting
  • expand upon max_speed/TDAT_refresh_rate relationship
  • add configurable LED’s per meter as different strip options impact distance/LED_spacing
  • completely revised WLED UDP Send

The YAML

esphome:
  name: olimex-esp32-poe
  on_boot:
    priority: 700
    then:
      - wait_until: 
            api.connected:
      - script.execute: get_all_params_script
      - switch.template.publish:
          id: detection_enabled
          state: ON
      - switch.template.publish:
          id: enable_logging
          state: OFF          

  includes:
    - kld7.h
    
esp32:
  board: esp32-poe-iso
  framework:
    type: arduino

substitutions:
  device_name: olimex-esp32-poe

button:
  - platform: restart
    name: Restart $device_name  
  
  - platform: template
    name: factory_reset_kld7 # RFSE
    id: factory_reset_kld7
    on_press:
      - uart.write: [0x52, 0x46, 0x53, 0x45, 0x00, 0x00, 0x00, 0x00]
      - delay: 2s
      - uart.write: [0x49,0x4E,0x49,0x54,0x04,0x00,0x00,0x00,0x00,0x00,0x00,0x00] # perform kld7 INIT

  - platform: template
    name: get_all_params # GRPS
    id: get_all_params
    on_press:
      - script.execute: get_all_params_script

script:
  - id: get_all_params_script
    then:
      - uart.write: [0x47, 0x52, 0x50, 0x53, 0x00, 0x00, 0x00, 0x00]

web_server:
  port: 80
  version: 2
  include_internal: true
  ota: false

# Enable logging
logger:
  # baud_rate: 0 # disable logging over uart
  logs:
    sensor: ERROR

# Enable Home Assistant API
api:

mdns:
  disabled: false

ota:
  password: "yadda"

ethernet:
  type: LAN8720
  mdc_pin: GPIO23
  mdio_pin: GPIO18
  clk_mode: GPIO17_OUT
  phy_addr: 0
  power_pin: GPIO12

switch:
  - platform: safe_mode
    name: use_safe_mode $device_name

  - platform: template
    name: enable_logging
    id: enable_logging
    optimistic: true

  - platform: template
    name: detection_enabled
    id: detection_enabled
    optimistic: true
    restore_state: true
    turn_on_action:
      - uart.write: [0x49,0x4E,0x49,0x54,0x04,0x00,0x00,0x00,0x00,0x00,0x00,0x00] # perform kld7 INIT
      - logger.log: "INIT sent"
    turn_off_action:
      - uart.write: [0x47, 0x42, 0x59, 0x45, 0x00, 0x00, 0x00, 0x00] # perform kld7 GBYE

number:
  - platform: template
    name: WLED_return_to_normal
    id: WLED_return_to_normal
    min_value: 1
    max_value: 15
    initial_value: 1
    step: 1
    restore_value: true
    optimistic: true

  - platform: template
    name: LEDs_per_Meter
    id: LEDs_per_Meter
    min_value: 30
    max_value: 144
    initial_value: 60
    step: 1
    restore_value: true
    optimistic: true
    mode: box

  - platform: template
    name: distance_offset
    id: distance_offset
    min_value: -255
    max_value: 255
    initial_value: 0
    step: 1
    restore_value: true
    unit_of_measurement: cm
    optimistic: true
    mode: box

  - platform: template
    name: max_speed # RSPI 0 = 12.5km/h, 1 = 25km/h, 2 = 50km/h, 3 = 100km/h
    id: max_speed
    min_value: 0
    max_value: 3
    initial_value: 1
    step: 1
    restore_value: true
    unit_of_measurement: kph
    set_action:
      - uart.write: !lambda
          return {0x52, 0x53, 0x50, 0x49, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: max_range # RRAI 0 = 5m, 1 = 10m, 2 = 30m, 3 = 100m
    id: max_range
    min_value: 0
    max_value: 3
    initial_value: 1
    step: 1
    restore_value: true
    set_action:
      - uart.write: !lambda
          return {0x52, 0x52, 0x41, 0x49, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: threshold_offset # THOF 10-60 dB
    id: threshold_offset
    min_value: 10
    max_value: 60
    initial_value: 30
    step: 1
    restore_value: true
    unit_of_measurement: dB
    set_action:
      - uart.write: !lambda
          return {0x54, 0x48, 0x4F, 0x46, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: tracking_filter # TRFT 0 = Standard, 1 = Fast detection, 2 = Long visibility
    id: tracking_filter
    min_value: 0
    max_value: 2
    initial_value: 0
    step: 1
    restore_value: true
    set_action: 
      - uart.write: !lambda
          return {0x54, 0x52, 0x46, 0x54, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: vibration_suppression # VISU 0-16, 0 = No suppression, 16 = High suppression
    id: vibration_suppression
    min_value: 0
    max_value: 16
    initial_value: 2
    step: 1
    restore_value: true
    set_action:
      - uart.write: !lambda
          return {0x56, 0x49, 0x53, 0x55, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: min_detection_dist # MIRA
    id: min_detection_dist
    min_value: 0
    max_value: 100
    initial_value: 0
    step: 1
    restore_value: true
    unit_of_measurement: "%"
    mode: box
    set_action:
      - uart.write: !lambda
          return {0x4d, 0x49, 0x52, 0x41, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};
          
  - platform: template
    name: max_detection_dist # MARA
    id: max_detection_dist
    min_value: 0
    max_value: 100
    initial_value: 50
    step: 1
    restore_value: true
    unit_of_measurement: "%"
    mode: box
    set_action:
      - uart.write: !lambda
          return {0x4d, 0x41, 0x52, 0x41, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: min_detection_angle # MIAN -90° to +90°
    id: min_detection_angle
    min_value: -90
    max_value: 90
    initial_value: -90
    step: 1
    restore_value: true
    unit_of_measurement: "°"
    mode: box
    set_action:
      - uart.write: !lambda
          return {0x4d, 0x49, 0x41, 0x4e, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: max_detection_angle # MAAN -90° to +90°
    id: max_detection_angle
    min_value: -90
    max_value: 90
    initial_value: 90
    step: 1
    restore_value: true
    unit_of_measurement: "°"
    mode: box
    set_action:
      - uart.write: !lambda
          return {0x4d, 0x41, 0x41, 0x4e, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: min_detection_speed # MISP 0 – 100% of speed setting
    id: min_detection_speed
    min_value: 0
    max_value: 100
    initial_value: 0
    step: 1
    restore_value: true
    unit_of_measurement: "%"
    mode: box
    set_action:
      - uart.write: !lambda
          return {0x4d, 0x49, 0x53, 0x50, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: max_detection_speed # MASP 0 – 100% of speed setting
    id: max_detection_speed
    min_value: 0
    max_value: 100
    initial_value: 100
    step: 1
    restore_value: true
    unit_of_measurement: "%"
    mode: box
    set_action:
      - uart.write: !lambda
          return {0x4d, 0x41, 0x53, 0x50, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: detection_direction # DEDI 0 = Receding, 1 = Approaching, 2 = Both
    id: detection_direction
    min_value: 0
    max_value: 2
    initial_value: 2
    step: 1
    restore_value: true
    set_action:
      - uart.write: !lambda
          return {0x44, 0x45, 0x44, 0x49, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: range_threshold # RATH 0 – 100% of range setting
    id: range_threshold
    min_value: 0
    max_value: 100
    initial_value: 10
    step: 1
    restore_value: true
    unit_of_measurement: "%"
    mode: box
    set_action:
      - uart.write: !lambda
          return {0x52, 0x41, 0x54, 0x48, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};
          
  - platform: template
    name: angle_threshold # ANTH -90° to +90°
    id: angle_threshold
    min_value: -90
    max_value: 90
    initial_value: 0
    step: 1
    restore_value: true
    unit_of_measurement: "°"
    mode: box
    set_action:
      - uart.write: !lambda
          return {0x41, 0x4e, 0x54, 0x48, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: speed_threshold # SPTH 0 – 100% of speed setting
    id: speed_threshold
    min_value: 0
    max_value: 100
    initial_value: 50
    step: 1
    restore_value: true
    unit_of_measurement: "%"
    mode: box
    set_action:
      - uart.write: !lambda
          return {0x53, 0x50, 0x54, 0x48, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

  - platform: template
    name: hold # HOLD 0 – 7200s 484f4c44
    id: hold
    min_value: 1
    max_value: 7200
    initial_value: 1
    unit_of_measurement: sec
    step: 1
    restore_value: true
    mode: box
    set_action:
      - uart.write: !lambda
          return {0x48, 0x4f, 0x4c, 0x44, 0x04, 0x00, 0x00, 0x00, (unsigned char)x, 0x00, 0x00, 0x00};

sensor:
  - platform: custom
    lambda: |-
      auto s = new kld7(id(uart_bus0));
      s->ip = "192.168.0.119";
      s->port = 21324;
      App.register_component(s);
      return {s->distance_sensor, s->speed_sensor, s->angle_sensor, s->db_sensor};
      
    sensors:
    - name: distance_sensor
      id: distance_sensor
      internal: true
      unit_of_measurement: cm
    - name: speed_sensor
      id: speed_sensor
      internal: true
      unit_of_measurement: kph
    - name: angle_sensor
      id: angle_sensor
      internal: true
      unit_of_measurement: deg
    - name: db_sensor
      id: db_sensor
      internal: true
      unit_of_measurement: dB

uart:
  id: uart_bus0
  rx_pin: GPIO36
  tx_pin: GPIO4
  baud_rate: 115200
  parity: EVEN

binary_sensor:
  - platform: gpio
    name: kld7_detection
    id: kld7_detection
    pin:
      number: GPIO05
      mode: INPUT_PULLDOWN
    on_state:
      then:
        while:
          condition:
            - binary_sensor.is_on: kld7_detection
            - switch.is_on: detection_enabled
          then:
            - uart.write: [0x47,0x4E,0x46,0x44,0x04,0x00,0x00,0x00,0x08,0x00,0x00,0x00] # Get TDAT
            - delay: !lambda
                if (id(max_speed).state == 0) {
                  return 250;
                } else if (id(max_speed).state == 1) {
                  return 125;
                } else if (id(max_speed).state == 2) {
                  return 65;
                } else if (id(max_speed).state == 3) {
                  return 35;
                } else {
                  return 250;
                }

kld7.h

#include "esphome.h"
#include "WiFiUdp.h"
#include <sstream>

class kld7 : public Component, public UARTDevice
{
public:
    std::string ip;
    uint16_t port;
    kld7(UARTComponent *parent) : UARTDevice(parent) {}

    Sensor *distance_sensor = new Sensor();
    Sensor *speed_sensor = new Sensor();
    Sensor *angle_sensor = new Sensor();
    Sensor *db_sensor = new Sensor();
    WiFiUDP UDP;

    std::vector<uint8_t> bytes;

    uint16_t oldLedOnIdx;
    std::vector<uint16_t> oldLedsOn;

    uint16_t newLedOnIdx;
    std::vector<uint16_t> newLedsOn;

    uint8_t onPacket[7] = {4, 1, 0, 0, 255, 5, 255};
    uint8_t offPacket[7] = {4, 1};

    bool doesHeaderMatch(std::vector<uint8_t> bytes, std::string header, int payloadLength)
    {
        if (bytes[0] != header[0] || bytes[1] != header[1] || bytes[2] != header[2] || bytes[3] != header[3])
        {
            return false;
        }

        return bytes[4] == payloadLength && bytes[5] == 0 && bytes[6] == 0 && bytes[7] == 0;
    }

    std::vector<uint16_t> fillVector(uint16_t firstIdx, uint16_t secondIdx)
    {
        uint16_t temp;

        if (firstIdx > secondIdx)
        {
            temp = firstIdx;
            firstIdx = secondIdx;
            secondIdx = temp;
        }

        std::vector<uint16_t> result;
        for (int i = firstIdx; i <= secondIdx; i++)
        {
            result.push_back(i);
        }

        return result; // always ascending
    }

    void setLedState(uint16_t index, bool state)
    {
        uint8_t *packet;
        packet = state ? onPacket : offPacket;

        packet[1] = (uint8_t)id(WLED_return_to_normal).state; //RTN
        packet[2] = (uint8_t)(index >> 8);
        packet[3] = (uint8_t)index;

        IPAddress ipaddr;
        ipaddr.fromString(ip.c_str());
        UDP.beginPacket(ipaddr, port);
        UDP.write(packet, 7);
        UDP.endPacket();
    }

    std::vector<uint16_t> difference(std::vector<uint16_t> target, std::vector<uint16_t> reference, bool reverse)
    {
        std::vector<uint16_t> diff;

        // make sure the vector is ascending. This only works because the vector is already sorted
        if (target.size() > 1 && target[1] < target[0])
        {
            std::reverse(target.begin(), target.end());
        }

        if (reference.size() > 1 && reference[1] < reference[0])
        {
            std::reverse(reference.begin(), reference.end());
        }

        std::set_difference(reference.begin(), reference.end(), target.begin(), target.end(), std::back_inserter(diff));

        if (reverse)
        {
            std::reverse(diff.begin(), diff.end());
        }

        return diff;
    }

    void loop() override
    {
        // ESP_LOGV("custom", "There is no byte: %i", bytes.size());
        while (available() > 0)
        {
            bytes.push_back(read());
            // ESP_LOGD("custom", "Show me the bytes: %i", bytes.size());
            if (bytes.size() < 8)
            {
                continue;
            }
            // ESP_LOGD("custom", "Size of bytes: %i", bytes.size()); // whelp..
            if (doesHeaderMatch(bytes, "TDAT", sizeof(TDAT)))
            {
                // ESP_LOGD("custom", "Size of TDAT: %i", sizeof(TDAT));
                // ESP_LOGD("custom", "Size of TDAT bytes: %i", bytes.size());
                if (bytes.size() < 8 + sizeof(TDAT))
                {
                    continue;
                }

                TDATUnion tdatUnion;
                std::copy(bytes.begin() + 8, bytes.end(), tdatUnion.bytes);
                // ESP_LOGD("custom", "Sending TDAT");
                distance_sensor->publish_state(tdatUnion.tdat.distance);
                speed_sensor->publish_state(tdatUnion.tdat.speed / 100);
                angle_sensor->publish_state(tdatUnion.tdat.angle / 100);
                db_sensor->publish_state(tdatUnion.tdat.db / 100);

                newLedOnIdx = (id(LEDs_per_Meter).state / 100) * (tdatUnion.tdat.distance - id(distance_offset).state);

                //momentum
                /*
                uint16_t diffLength = newLedOnIdx - oldLedOnIdx;
                if (diffLength < -3 || diffLength > 3)
                {
                    int newMomentumIdx = newLedOnIdx + (int)(diffLength * 0.5);
                    if (newMomentumIdx < 0)
                    {
                        newLedOnIdx = 0;
                    }
                    else
                    {
                        newLedOnIdx = newMomentumIdx;
                    }
                }  
                */
                newLedsOn = fillVector(oldLedOnIdx, newLedOnIdx);

                // get direction of travel
                bool movingRtl = newLedOnIdx < oldLedOnIdx;

                std::vector<uint16_t> toTurnOff = difference(newLedsOn, oldLedsOn, movingRtl);
                std::vector<uint16_t> toTurnOn = difference(oldLedsOn, newLedsOn, movingRtl);

                for (auto offIdx : toTurnOff)
                {
                    setLedState(offIdx, false);
                }

                for (auto onIdx : toTurnOn)
                {
                    setLedState(onIdx, true);
                }

                oldLedOnIdx = newLedOnIdx;
                oldLedsOn = newLedsOn;

                bytes.clear();
            }
            else if (doesHeaderMatch(bytes, "RPST", sizeof(RPST)))
            {
                // ESP_LOGD("custom", "Size of RPST: %i", sizeof(RPST));
                // ESP_LOGD("custom", "Size of RPST bytes: %i", bytes.size());
                if (bytes.size() < 8 + sizeof(RPST))
                {
                    continue;
                }

                RPSTUnion rpstUnion;
                std::copy(bytes.begin() + 8, bytes.end(), rpstUnion.bytes);
                // ESP_LOGD("custom", "Sending RPST");
                id(max_speed).publish_state(rpstUnion.rpst.max_speed);
                id(max_range).publish_state(rpstUnion.rpst.max_range);
                id(threshold_offset).publish_state(rpstUnion.rpst.threshold_offset);
                id(tracking_filter).publish_state(rpstUnion.rpst.tracking_filter);
                id(vibration_suppression).publish_state(rpstUnion.rpst.vibration_suppression);
                id(min_detection_dist).publish_state(rpstUnion.rpst.min_detection_dist);
                id(max_detection_dist).publish_state(rpstUnion.rpst.max_detection_dist);
                id(min_detection_angle).publish_state(rpstUnion.rpst.min_detection_angle);
                id(max_detection_angle).publish_state(rpstUnion.rpst.max_detection_angle);
                id(min_detection_speed).publish_state(rpstUnion.rpst.min_detection_speed);
                id(max_detection_speed).publish_state(rpstUnion.rpst.max_detection_speed);
                id(detection_direction).publish_state(rpstUnion.rpst.detection_direction);
                id(range_threshold).publish_state(rpstUnion.rpst.range_threshold);
                id(angle_threshold).publish_state(rpstUnion.rpst.angle_threshold);
                id(speed_threshold).publish_state(rpstUnion.rpst.speed_threshold);
                id(hold).publish_state(rpstUnion.rpst.hold);

                bytes.clear();
            }
            else
            {
                bytes.erase(bytes.begin());
                continue;
            }
        }
    }

    typedef struct
    {
        char version[19];              // Software Version STRING 19 Zero-terminated String K-LD7_APP-RFB-XXXX
        uint8_t base_frequency;        // Base frequency UINT8 1 0 = Low, 1 = Middle, 2 = High 1 = Middle
        uint8_t max_speed;             // Maximum speed UINT8 1 0 = 12.5km/h, 1 = 25km/h, 2 = 50km/h, 3 = 100km/h 1 = 25km/h
        uint8_t max_range;             // Maximum range UINT8 1 0 = 5m, 1 = 10m, 2 = 30m, 3 = 100m 1 = 10m
        uint8_t threshold_offset;      // Threshold offset UINT8 1 10-60 dB 30 dB
        uint8_t tracking_filter;       // Tracking filter type UINT8 1 0 = Standard, 1 = Fast detection, 2 = Long visibility 0 = Standard
        uint8_t vibration_suppression; // Vibration suppression UINT8 1 0-16, 0 = No suppression, 16 = High suppression 2 = Low suppression
        uint8_t min_detection_dist;    // Minimum detection distance UINT8 1 0–100% of range setting 0%
        uint8_t max_detection_dist;    // Maximum detection distance UINT8 1 0–100% of range setting 50%
        int8_t min_detection_angle;    // Minimum detection angle INT8 1 -90° to +90° -90°
        int8_t max_detection_angle;    // Maximum detection angle INT8 1 -90° to +90° +90°
        uint8_t min_detection_speed;   // Minimum detection speed UINT8 1 0–100% of speed setting 0%
        uint8_t max_detection_speed;   // Maximum detection speed UINT8 1 0–100% of speed setting 100%
        uint8_t detection_direction;   // Detection direction UINT8 1 0 = Receding, 1 = Approaching, 2 = Both 2 = Both
        uint8_t range_threshold;       // Range threshold UINT8 1 0–100% of range setting 10%
        int8_t angle_threshold;        // Angle threshold INT8 1 -90° to +90° 0°
        uint8_t speed_threshold;       // Speed threshold UINT8 1 0–100% of speed setting 50%
        uint8_t IO1;                   // Digital output 1 UINT8 1 0 =Direction, 1 = Angle, 2 = Range, 3 = Speed, 4 = Micro detection 0 = Direction
        uint8_t IO2;                   // Digital output 2 UINT8 1 0 =Direction, 1 = Angle, 2 = Range, 3 = Speed, 4 = Micro detection 1 = Angle
        uint8_t IO3;                   // Digital output 3 UINT8 1 0 =Direction, 1 = Angle, 2 = Range, 3 = Speed, 4 = Micro detection 2 = Range
        uint16_t hold;                 // Hold time UINT16 2 1–7200s (1s–2h) 1s
        uint8_t micro_retrigger;       // Micro detection retrigger UINT8 1 0 = Off, 1 = Retrigger 0 = Off
        uint8_t micro_sensitivity;     // Micro detection sensitivity UINT8
    } RPST;

    typedef union
    {
        RPST rpst;
        uint8_t bytes[sizeof(RPST)];
    } RPSTUnion;

    typedef struct
    {
        uint16_t distance;
        int16_t speed;
        int16_t angle;
        uint16_t db;
    } TDAT;

    typedef union
    {
        TDAT tdat;
        uint8_t bytes[sizeof(TDAT)];
    } TDATUnion;
};
1 Like

We need a video of the lights following you! The dremled-off antenna is the cherry on top.

Can’t wait for the turret. I could use something to repel the rabbits from my yard…

2 Likes

Super cool project! I’ve been following all these DIY mmWave projects for the past couple weeks and wanted to try my hand at building one myself. I am curious if anyone has compiled a list of modules with pros and cons? I want to buy one to mess around. Any suggestions?

I’m definitely not qualified to make the list, but from reading here’s what I’ve come to understand:

  • DFRobot SEN0395 ($29)
    • Simple GPIO interface and easy to setup for boolean information
    • Super sensitive
    • Successfully used in mmWave Presence Detection
  • RFBeam K-LD7 ($86)
    • Requires custom UART interface
    • More capable module with speed, angle, and tracking capabilities
    • Used in this project
    • Performance still to be seen
  • RFBeam K-LD2 ($53)
    • Simple GPIO interface limits the usability of its extra features (speed, area, etc)
  • SeeedStudio Human Stationary Presence Module ($20)
    • Usable but has issues with seeing through walls
    • Doesn’t seem to work well (post)

Other sensors that haven’t been tried:

  • Jorjin MM5D91
    • Datasheet seems promising
    • However, it needs software to reconfigure (no documented UART protocol for this)
  • SeeedStudio 60GHz mmWave Radar Sensor - Fall Detection Module Pro
    • Configurable max range
    • Claimed to be more accurate
2 Likes
  • DFRobot SEN0395 ($29)

    • It’s good :+1:
    • Detects static humans (sleeping, etc.)
    • Good luck getting one since Everything Smart Home made a vid.
  • RFBeam K-LD7 ($86)

    • Doesn’t detect static humans for very long.
    • Mainly it’s just great for tracking people’s movement.
    • 2D not 3D
  • RFBeam K-LD2 ($53)

    • A less expensive worse LD7. Enough for some use cases.
  • SeeedStudio 24GHz Human Stationary Presence Module ($20)
    (similar to the 24GHz Fall Detection Module)

    • No max distance setting.
    • Has a very hard time returning to an unoccupied state depending on the environment. I suspect the unconfigurable, huge 16.5m range is the culprit.
    • Actually decent as a “detect-how-much-motion” sensor. Gives a range from 0-100. Someone make a virtual trainer that yells at you to start moving.
  • SeeedStudio 60GHz Fall Detection Pro ($37)

    • It could suffer similar issues as 24GHz since sensors which return a 0 or 1 (unoccupied / occupied) are at the whim of the internal processing algorithm. Same company same algorithms?
    • 60GHz is more accurate on a theoretical level. Has a max distance setting. Uses fmcw technology instead of dopper. These are all promising signs.
    • Not for the faint of heart to be an early adopter. Send me one and I’ll pay with example code :wink:
    • 60GHz does not penetrate walls as much as 24GHz does (good or bad)
    • Shipping from Seeed is expensive. Digikey will get them Aug 20. I might get one in August.
  • SeeedStudio 60GHz Respiratory Heartbeat Sensor ($45)

  • Jorjin MM5D91-00

    • Has an occupancy gpio (High / Low) so if you can get it configured with their software, the setting is saved on the chip and you could then read the gpio with an esp.
    • Based on an Infineon 60Ghz mmWave radar chip which is the same as the SeeedStudio 60GHz Modules. This does not mean the uart protocol is the same nor does it necessarily mean the processing algorithm is the same?
    • Definitely seems workable on paper.
1 Like

Gotta make it pretty first! LED strip is currently strung up on the wall between pictures, school art, a foam makeup head; you know, normal things…

And excellent synopsis of the sensors in question. My biggest concern w/ the 60Ghz Seeedstudio is that the “Universal Protocol” they advertise is all very high-level interpretive results compared to lower-level raw data formats.

GIF added to OP.

1 Like

Close In LEGO System…

4 Likes

New plan. No more WLED, ready for the Nerf…

Hi @crlogic

What are you using GPIO05 for here?

That is target detection.

Hi @crlogic

In your initial post, regarding creating the sensors, you haven’t specified

      s->ip = "192.168.0.119";
      s->port = 21324;

What are the ip pointing to? The ESPHome device or ESPHome server (HA) ?
I assume this is the reason I don’t get the sensors listed in HA…

Oh yeah, good catch. The original version of this project made an WLED-based individually addressable LED strip “follow-me” as I walked by. So that IP/port was the WLED device. The mish-mash of the OP post is because scope-creep. I realized there was so much more that I could use a “target-tracking” radar for. Albeit it would be cool to modify it so that when a car drives up the lights do different things :wink:

Yes this is deliberately avoided since the ESP is doing all the work and HA would really only be used as a “control-plane” to turn functions on/off.

I am currently trying to figure out how to best mount a semi-auto Nerf and cycle the cylinder. I have a servo for that, I have a relay for the motor. Now I have physical design challenges…

I would actually like to have these sensors created in HA… At least for playing with use-cases :slight_smile: SO I changed internal: true to false, the sensor appear but only with unknown state?

I’ve also done some testing with the Seeed 60GHz Fall Sensor. It is not even close to be ready, important functions are still not implemented…:

  • Range
  • I can not get fall detection to work for some reason
  • Do get quickly back to ‘idle’/0 after no detection

It do quite good Occupied, Stationary, None though, but still need to play a bit with it…

Best to post your full code to see what is going on…

I think that SeeedStudio have taken a very narrow approach to their sensor implementation. And unless you fit their specific use-case it will be challenging to use as a “general purpose sensor.”

I recall looking at the “Breathing/Heart Rate” version and recall that is only had a range of 2M ? That doesn’t sound very useful for general purpose occupancy which it seems that the majority of HA users want.

So it is good to hear your report on the Fall version as they come from the same family and are by no means inexpensive.

Actually just using yours - only replaced the UART GPIO’s and using WIFI rather than ETH.

sensor:
  - platform: custom
    lambda: |-
      auto s = new kld7(id(uart_bus0));
      App.register_component(s);
      return {s->distance_sensor, s->speed_sensor, s->angle_sensor, s->db_sensor};
      
    sensors:
    - name: distance_sensor
      id: distance_sensor
      internal: false
      unit_of_measurement: cm
    - name: speed_sensor
      id: speed_sensor
      internal: true
      unit_of_measurement: kph
    - name: angle_sensor
      id: angle_sensor
      internal: false
      unit_of_measurement: deg
    - name: db_sensor
      id: db_sensor
      internal: true
      unit_of_measurement: dB
      

Seems the return do not provide any value ?
You see I’ve added Angle and Distance to HA.

what if there’s two people in the room?

detection_enabled is on? You have UART configured RX–>TX & TX–>RX ?

When testing, you could enable UART debugging to ensure the ESP is seeing the target data from the K-LD7.

  - id: uart_bus_kld7
    rx_pin: GPIO26
    tx_pin: GPIO33
    baud_rate: 115200
    parity: EVEN
    debug:
      direction: RX
      after:
        delimiter: [0x52, 0x45, 0x53, 0x50, 0x01, 0x00, 0x00, 0x00, 0x00] 
      sequence:
        - lambda: UARTDebug::log_hex(direction, bytes, ':');

Good question. The sensor selects its “single target” based on greatest “point cloud” aka, biggest signal(s) detected.

So if the two people are of equal “size” (height/weight/etc) then it will focus on the one closest.

If they are equal distance, but one is moving more, that means more target “points” so it will turn to them.

It boils down to radar RCS (Radar Cross Section). Aka, the relationship between size and distance. So a small child that is closer to the radar might be a smaller RCS than a large adult slightly further away. The radar “sees” RCS and the largest wins.

As I can enable/disable detection_enabled, I know RX-TX are good (ESP32 actually refuses to boot if they’re wrong).

Where should I put

Under logger: ?