Reading encrypted udp component sensors in node red

[aka47]

Guys

I am trying to get some UDP component sensors read into local node red instances and the broadcast nature of this component is really useful for this.

I have a dev ESP32 device set up fine using esphome, it talks to HA fine, and it broadcasts UDP fine as well. My node red instances can also see the UDP packets, capture them and poke them into flows as well. So far so good and very promising.

I have mangled up a bunch of node red code and can parse the header and encrypted payload fine too. What I am struggling with is decrypting the payload contents.

So far I have 3 nodes, one that parses the header into its plaintext and encrypted payload components (Parse header), Once that decrypts the encrypted payload (Decrypt payload) and am working on the last part that parses the decrypted payload into a message per sensor.

I have done a first stab at this using some code I found at:-

XXTEA

javascript implementation
https://www.movable-type.co.uk/scripts/tea-block.html

But the decrypt does not seem to be happening, at least there is nothing that resembles the sensor name etc in the decrypted data.

Questions:-

1. How is the key being hashed in the component, I have looked through the source and am finding it a touch obscure.

2. Is there anyone that contributed to this component that would be prepared to spend a small amount of time helping me get this last part nailed down.

If I can get this working I would be happy to donate the resulting node.js somewhere or other to help others do similar.

Cheers

aka47

[clydebarrow (Clyde Stubbs)]

I’d suggest you look at the source code in ESPHome to work from. The XXTEA implementation uses a 256 bit key instead of the original 128 bit, so using code from elsewhere isn’t going to work without modification.

[aka47]

Clyde thanks for taking the time to reply.

I had noticed the difference in key lengths and modified the node.js code to suit. This was pretty much the first thing after I compared it with the ESPhome udp component source and chose it as a start point. Standing on the shoulders of giants and all that.

I am now pretty much at the debug the code/process point. Starting at the beginning I am keen to pin down that I have got the correct feed in data, treating the decrypt function as a black box until later. Crap in = Crap out being the usual maxim.

So the focus at the moment is on the key. ESPHome’s documentation states that the key is 256bit hashed. But I am struggling to see a hashing function or it’s application in the source. Hence my comment about it being a touch obscure.

Maybe it is’nt hashed after all but is only a byte copy of the key truncated/null-filled. Part of the issue with reading C++ source is that the data hiding/encapsulating nature of object oriented programming can hide a lot stuff, without it being apparent that it has done so.

From a computing science perspective I would be struggling to describe a byte copy of a key with truncate/null-filling length matching as hashing. But there you go, maybe I am being too pedantic.

Whichever way it is being done I need to match it in the node.js code if any attempt at further decryption is expected to ever succeed. Irrespective of the contents of the decrypt black box.

Once I am reasonably sure that the crypt-text and (hashed??) decrypt key are a perfect match for how it would be done by an ESPHome instance on the decrypt I can progress to the rest of the processing/decrypt pipeline/code.

Cheers

aka47

[clydebarrow (Clyde Stubbs)]

@aka47

But I am struggling to see a hashing function or it’s application in the source.

esphome/esphome/blob/cf835d15806481c50361cd6d8d193cc7b0b0175e/esphome/components/udp/__init__.py#L121

[aka47]

Yup that will do it, thanks for the pointer. I will have a rummage and see what node.js I can use to match the function. Or maybe generate a prehashed key using python to feed into the node.js pre-chewed.

I guess it helps if I look where the hashing was done (Python) rather than in the C++ code lol.

Cheers

aka47

[aka47]

Last quick question for now. Being as yet reasonably unfamiliar with the intricacies of ESPHome builds. In terms of generating a prechewed key.

is:-

list(hashlib.sha256(config[CONF_KEY].encode()).digest())

doing something appreciably different from :-

list(hashlib.sha256(b"SomeKeyHere").digest())

Cheers

aka47

[clydebarrow (Clyde Stubbs)]

Not an ESHPhome question:

Unicode HOWTO

[aka47]

Ok after trying a bunch of stuff we still can not get a useful decrypt.

I tried prechewing the key, it looks like it should be correct but there is no way that I can see of easily verifying the results though as HA/ESPhome don’t show you anywhere what they have produced as the key.

ANyways I tried with the modified JS I had culled from t’internet and then tried recoding/porting the decrypt function from the ESPHome UDP component. With no useful result as yet.

A copy of the decrypt node red function node code

// https://en.wikipedia.org/wiki/XXTEA
// https://github.com/esphome/esphome/tree/dev/esphome/components/udp

// useful variables etc
var cryptarray = new Uint32Array(msg.payload);
var keyarray = new Uint32Array(8);
var arrayindex = 0;
var bufindex = 0;

// prechewed decrypt key created using python eg
// import hashlib
// mykey = "my key here"
// list(hashlib.sha256(mykey.encode()).digest())
var keybuf = Buffer.from([156, 223, 29, 168, 210, 239, 170, 184, 155, 65, 237, 206, 220, 7, 39, 100, 76, 191, 63, 149, 105, 248, 204, 80, 243, 63, 120, 92, 229, 125, 141, 189]);

// fill the keyblock using key
for(arrayindex = 0; arrayindex < 8; arrayindex++) {
    keyarray[arrayindex] = keybuf.readUInt32LE(bufindex);
    bufindex +=4;
}

//uncomment for debuging
//msg.cryptarray = msg.payload;
//msg.keyarray = keyarray;
//msg.key = keybuf;

// Definition of function to decrypt
// XXTEA implementation, using 256 bit key.
// 256 bit key organised as uint32 array, 32 bytes or 8 uint32 words
//  v is n-word data vector converted to array of uint32
//  k is 8-word key converted to array of uint32
//  * XXTEA: decodes array of unsigned 32-bit integers using 256-bit key.
//  * @param   {number[]} v - Data vector.
//  * @param   {number[]} k - Key.
function decode(v,k) {
   var delta = 0x9e3779b9;
   var mx;
   var z;
   var y = v[0];
   var e;
   var p;
   var n = v.length;
   var q = Math.floor(6 + 52 / n);
   var sum = q * delta;

   while(--q != 0) {
      e = (sum >>> 2);
      for( p = n - 1; p != 0; p--) {
        z = v[p - 1];
        mx = ((((z >>> 5) ^ (y << 2)) + ((y >>> 3) ^ (z << 4))) ^ ((sum ^ y) + (k[(p ^ e) & 7] ^ z)));
        y = v[p] -= mx;
      }
      z = v[n - 1];
      mx = ((((z >>> 5) ^ (y << 2)) + ((y >>> 3) ^ (z << 4))) ^ ((sum ^ y) + (k[(p ^ e) & 7] ^ z)));      
      y = v[0] -= mx;
      sum -= delta;
   }
}

// The actual decrypt, NB decrypt happens in place
decode(cryptarray, keyarray);
msg.payload = cryptarray;
msg.topic = "TEAXX Decrypt"

return msg;

[clydebarrow (Clyde Stubbs)]

@aka47

var cryptarray = new Uint32Array(msg.payload);

This looks odd. This will only do what you want if msg.payload is already a Uint32Array which I would not expect. And if it is, why copy it?

[clydebarrow (Clyde Stubbs)]

@aka47

though as HA/ESPhome don’t show you anywhere what they have produced as the key.

Nothing to stop you testing the same code in Python. The other way is to work with a git clone of ESPHome, run from the command line, and add print() in key places in the Python code.

[aka47]

The payload is already a Uint32Array I converted it from the UDP byte packet (buffer) in the previous node in the flow.

In NodeRed lumps of data is passed between nodes (it flows) as messages. Messages are typically, JSON assemblages of items, but can be just a single item.

In the case of my solution I have 3 function nodes and a node red sorting node. The Function nodes are “Parse Header”, “XXTEA Decrypt”, “Parse Payload”. the node red sorting node is just a message router for what comes out of “Parse Payload”.

I think I will avoid stating that this is’nt ESPHome and putting a link in here to node.js and or Node Red messages etc. But explain instead as it is useful and tangentially relevant. Plus I guess it is a helpful and friendly thing to do. I don’t expect every one I speak to to know everything about every language, or engage in unnecessary yak shaving.

msg.payload is not the actual payload it is a reference to the contents of the payload. item in the input message. If I want to be able to make sure I have an unaltered copy of the same contents for alter use, ie debugging I need to make a new copy of the contents of msg.payload.

var cryptarray = new Uint32Array(msg.payload);

creates the variable cryptarray, which refers to a new object of type Uin32Array which has the contents of msg.payload copied to it.

This is particularly important if we want do do stuff like

//uncomment for debuging
//msg.cryptarray = msg.payload;
//msg.keyarray = keyarray;
//msg.key = keybuf;

Debugging a message stream along it’s processing pipeline is aided by accumulating stuff in the message I can directly compare at different stages in their processing pipeline life cycle. Once I have got a workable solution I can go back and comment out the bits that add stuff to the message that are unnecessary for operation.

Doing :-

var cryptarray = msg.payload;
var cleararray = msg.payload;

Gives me two references to the same object, changing one changes both which is not what is required. You will end up with two items in your msg that have different names but are apparently identical in contents. Pretty useless for debugging.

Hope this helps

Cheers

aka47

[aka47]

See other referrence to yak shaving.

If when everyone needed to put my coffee down on a desk so they could write code, they had to learn arboriculture, grow trees, learn sawyering, joinery, finishing and then apply it to building said desk. No Code would ever get written. We can often forget as coders/creatives how many shoulders we stood upon to see just that bit further in achieving our objectives, and deny others the same opportunities through our carelessness.

[aka47]

A piccy for illustrative purposes. Piccy’s being much better than my error prone dyslexic text.

[aka47]

The unworking (not decrypting) code I pasted up is solely from the TEAXX Decrypt function node.

Maybe later when we have a working base to push out from I will refactor and condense it into one function block and add in more clever stuff like ping/pong etc

For now a simple basic decrypt (no rolling codes, no ping/pong etc) is all that is required. The rest can follow later.