How to read and parse UART HEX data?

crlogic · February 10, 2022, 6:36pm

I am in a little above my head and hope someone can nudge me in the right direction.

Goal: read and parse hex data from a uart.

Challenge: working with hex from a uart

The specifics:

Following page 19 of the devices documentation reference I can successfully initialize the device

- uart.write: [0x49,0x4E,0x49,0x54,0x04,0x00,0x00,0x00,0x00,0x00,0x00,0x00]

Using the GPIO binary_sensor to trigger a request for data I can see the data I want

binary_sensor:
  - platform: gpio
    name: kld7_detection
    id: kld7_detection
    pin:
      number: GPIO23
      mode: INPUT_PULLDOWN
    on_state:
      then:
        while:
          condition:
            - binary_sensor.is_on: kld7_detection
          then:
            - uart.write: [0x47,0x4E,0x46,0x44,0x04,0x00,0x00,0x00,0x08,0x00,0x00,0x00] # Get TDAT
            - delay: 100ms

I can see the hex values using uart_debug and am now running up to the wall of my knowledge of how to consume it. The next steps are clear from a logical perspective, eg, index array of hex values, then convert, etc. Table 15 of the docs tell me what each hex value is;

Welcoming all pointers/docs/samples for the next steps that I think I need to accomplish

how do I capture this uart output into a consumable variable/component?
– is my only option a Custom UART Device and the Arduino API? I don’t know C++ so this is a struggle beyond basic concepts.
index the returned data (byte array?) the results should always be in the same position unless there is no “detection”
combine two hex values and convert to float/int depending on value (distance,speed,angle,db)
return/publish the four different values so they can be consumed elsewhere

zoogara · February 10, 2022, 9:15pm

I think you are going to have to learn how to code.

The Custom UART Text Sensor page gives a pretty good example of reading through the UART input buffer.

You will need to read through the data stream and extract the data strings from the various positions, doing error checking for when no data returned. The data you retrieved will then need to be converted from hex string to numeric.

crlogic · February 10, 2022, 9:42pm

I have been playing with this and can’t quite find the best way to approach the parsing required.

The example assumes \r or \n which there are none (regularly) in the data stream. So that’s the first challenge.

The UARTDebug seems to read complete lines just fine though.

Is the only way to do this is learn cpp?

zoogara · February 10, 2022, 11:16pm

Pretty well.

/r and /n are shorthand for carriage return and line feed (newline).

At least one of them is probably in your data stream terminating a line of data.

crlogic · February 10, 2022, 11:26pm

I’ve tried both using the example from the docs (removed the case/break).

Would the UARTDebug string show it? As I see nothing for each line, just random ones.

The sensor datasheet shows response examples with delcared length return values. Would that negate a carriage return?

zoogara · February 10, 2022, 11:42pm

Post your whole YAML, not just the extracts. Don’t forget to remove any ids like wifi details.

zoogara · February 11, 2022, 12:35am

Maybe forget about doing this on on ESPHome and go for a Pi Zero? The manufacture has already supplied Python scripts that do what you want. You just need to add some MQTT script to get the data to wherever you want it.

https://forums.raspberrypi.com/viewtopic.php?t=317286

crlogic · February 11, 2022, 2:17am

My bad, I’ll post the full yaml tomorrow.

As for the python scripts, I did see that. My intentions were to leverage my last experience with ESPHome directly controlling WLED without HA for faster response times. While permitting the option to include HA and features such as OTA. The last being of particular value. If a full Pi is the answer then I would reluctantly go there. Albeit I’d likely try MicroPython first. The size of the sensor/MCU packaging is a consideration here.

crlogic · February 12, 2022, 6:09pm

As promised, yaml below. More sample data and observations.

Observations:

there is no newline or carriage return in the datastream. This makes the sample .h provided in the Custom UART Text Sensor not a copy/paste affair.
the Target Data (TDAT) provided by the radar varies in length. I believe this means that the returned length response will need to be read as it comes in and then adjust the number of bytes read for the payload.
I have been reading the Serial Input Basics - updated thread in order to learn how to leverage the cpp lambda options to extract the data but all examples provided are fixed start/end deliminators which sadly does not align with radar TX behavior.

ESPHome yaml below (have skipped all the normal api/wifi things…)

esphome:
  name: tinypico-kld7
  platform: ESP32
  board: esp32dev
  includes:
    - kld7_uart_sensor.h
  on_boot:
    priority: -100
    then:
      - uart.write: [0x49,0x4E,0x49,0x54,0x04,0x00,0x00,0x00,0x00,0x00,0x00,0x00] # perform kld7 INIT

# Enable logging
logger:

uart:
  id: uart_bus
  tx_pin: GPIO19
  rx_pin: GPIO18
  baud_rate: 115200
  parity: even
  debug:
    direction: BOTH
    # direction: RX
    dummy_receiver: false
    after:
      delimiter: "RESP\x01\x00\x00\x00\x00"
      bytes: 25
      timeout: 80ms
    sequence:
      # - lambda: UARTDebug::log_hex(direction, bytes, ':');
      - lambda: UARTDebug::log_string(direction, bytes);
      # - lambda: UARTDebug::log_int(direction, bytes, ',');
      # - lambda: UARTDebug::log_binary(direction, bytes, ';');

binary_sensor:
  - platform: gpio
    name: kld7_detection
    id: kld7_detection
    pin:
      number: GPIO23
      mode: INPUT_PULLDOWN
    on_state:
      then:
        while:
          condition:
            - binary_sensor.is_on: kld7_detection
          then:
            - uart.write: [0x47,0x4E,0x46,0x44,0x04,0x00,0x00,0x00,0x08,0x00,0x00,0x00] # Get TDAT
            - delay: 100ms
        
text_sensor:
- platform: custom
  lambda: |-
    auto my_custom_sensor = new UartReadLineSensor(id(uart_bus));
    App.register_component(my_custom_sensor);
    return {my_custom_sensor};
  text_sensors:
    id: "uart_hex"

Here is sample data returned from the radar

[12:50:24][D][uart_debug:158]: >>> "GNFD\x04\x00\x00\x00\b\x00\x00\x00"
[12:50:24][D][uart_debug:158]: <<< "RESP\x01\x00\x00\x00\x00"
[12:50:24][D][uart_debug:158]: <<< "TDAT\b\x00\x00\x00u\x00\xDC\xFF\t\xFB\x16"
[12:50:24][D][uart_debug:158]: >>> "GNFD\x04\x00\x00\x00\b\x00\x00\x00"
[12:50:24][D][uart_debug:158]: <<< "RESP\x01\x00\x00\x00\x00"
[12:50:24][D][uart_debug:158]: <<< "TDAT\b\x00\x00\x00s\x00\xD1\xFF\xB1\xF9\x1F\x16"
[12:50:24][D][uart_debug:158]: >>> "GNFD\x04\x00\x00\x00\b\x00\x00\x00"
[12:50:24][D][uart_debug:158]: <<< "RESP\x01\x00\x00\x00\x00"
[12:50:24][D][uart_debug:158]: <<< "TDAT\b\x00\x00\x00q\x00\xCE\xFF\xC0\xF8\x91\x15"
[12:50:24][D][uart_debug:158]: >>> "GNFD\x04\x00\x00\x00\b\x00\x00\x00"
[12:50:24][D][uart_debug:158]: <<< "RESP\x01\x00\x00\x00\x00"
[12:50:24][D][uart_debug:158]: <<< "TDAT\b\x00\x00\x00p\x00\xEA\xFFY\xF84\x16"
[12:50:24][D][uart_debug:158]: >>> "GNFD\x04\x00\x00\x00\b\x00\x00\x00"
[12:50:24][D][uart_debug:158]: <<< "RESP\x01\x00\x00\x00\x00"
[12:50:25][D][uart_debug:158]: <<< "TDAT\b\x00\x00\x00r\x00\x12\x00\xC2\xF6%\x15"
[12:50:25][D][uart_debug:158]: >>> "GNFD\x04\x00\x00\x00\b\x00\x00\x00"
[12:50:25][D][uart_debug:158]: <<< "RESP\x01\x00\x00\x00\x00"
[12:50:25][D][uart_debug:158]: <<< "TDAT\b\x00\x00\x00q\x00\x06\x00\x1E\xF7\xC1\x13"
[12:50:25][D][uart_debug:158]: >>> "GNFD\x04\x00\x00\x00\b\x00\x00\x00"
[12:50:25][D][uart_debug:158]: <<< "RESP\x01\x00\x00\x00\x00"
[12:50:25][D][uart_debug:158]: <<< "TDAT\b\x00\x00\x00n\x00\xD5\xFF\x14\xF8\x94\x14"

What I think I have to do it is try and incorporate the following steps in cpp functions

use the "RESP\x01\x00\x00\x00\x00" as a StartMarker
read the next eight bytes and byte eight should be the length that gets read for the payload
do payload parsing things (this scares me)
– splits every two bytes
– convert little endian to big endian (I assume)
– convert to decimal
– return an array of the three decimal values

petro · February 12, 2022, 7:21pm

I’m probably late to the thread, but in home assistant you can use a template to get the data you want. So if you can just get the hex codes as a binary string as the state you can use a template sensor to filter the data.

crlogic · February 12, 2022, 8:15pm

I think there is still the problem of having no newline/carriage_return. Since the assumption here is that the text_sensor publishes a value to HA and it only does so when it see’s a regular \n or \r which don’t exist in the data

Which brings up the second conflict, which I haven’t mentioned to date. I don’t actually want HA to be required. Yes obviously I am using ESPHome now, to develop, since I don’t write cpp. But I want to be able to move “off network” and have the radar be standalone until brought back. It will feed WLED via HTTP calls (maybe via UDP calls in the future).

The ultimate goal where is [standalone ESPHome radar data] → [WLED] where HA is only needed to OTA.

I think at this point I have come to realize that my question isn’t actually about ESPHome and I need to move to the Arduino forum.

zoogara · February 13, 2022, 12:12am

Quick one as I’m AFK for a few days, but if you look at your debug definition, you have a delimiter string that is a substitute for /r /n. Plug that into the example and go from there.

      delimiter: "RESP\x01\x00\x00\x00\x00"

crlogic · February 13, 2022, 12:32am

Yes I’ve tried that with no success. I suspect that the ESPHome interface between the yaml and Arduino layers performs some level of interpretation as that syntax doesn’t work in the cpp .h file.

Looking at the python example mentioned earlier, struct is doing all the work to translate into a bytearray (eg. No “RESP” chars).

Having successfully used the python library I may cave and go RPi Zero W just to get things working.

I think I have perhaps distilled the problem definition into, “how do you rewrite struct in C++?”

nickrout · February 13, 2022, 1:39am

Have you looked at the python implementation here? A driver for the K-LD7 radar module — kld7 0.1.1 documentation

The API is also documented here A driver for the K-LD7 radar module — kld7 0.1.1 documentation

Perhaps micropython or circuitpython would be a good way to get this onto a microcontroller?

crlogic · February 13, 2022, 1:50am

Sorry, that’s the one using struct I was looking at.

It’s funny because I first got started in MCU’s last year by using MicroPython. It was never the most reliable for me and then I found HA + ESPHome which has been great.

I think I’ll take baby steps. Get full OS + Python a go with that module. Then see if that repo can be massaged to work in MicroPython.

nickrout · February 13, 2022, 2:42am

I just started to use circuitpython for a little project, and it has been excellent, but as I say it was a very little project.

crlogic · February 13, 2022, 2:50am

CircuitPython is great too.

My challenge was dropping off wifi after a day or two. Which may have changed in the past year. I guess I’ll find out!

crlogic · February 18, 2022, 4:31pm

Massaged the python library into CircuitPython on an ESP32-S2 sending data to WLED via UDP Sync. Maybe one day I’ll know enough C++ to get it working in ESPHome.

Pretty basic POC using radar distance tracking: Radar tracking - Album on Imgur

Mahko_Mahko · April 21, 2022, 12:26pm

Dunno if it would help but have you tinkered with some of the other debug options?

Stuff like this…


uart:
  - id: uart_bus2
    tx_pin: GPIO17
    rx_pin: GPIO16
    debug:
      direction: BOTH
      dummy_receiver: true
      after:
        bytes: 4
      sequence:     
        - lambda: UARTDebug::log_int(direction, bytes, ',');

crlogic · April 21, 2022, 12:29pm

Thanks for the suggestion. The sensor in question does not use line/carriage returns of any sort. After further testing I can confirm that fixed size byte arrays are indeed what are used. Got it working with MicroPython and have yet to loop back to decipher the hex data in cpp. Thx!