Hex string to binary conversion issue

[Narb]

Hi,

I am trying to convert a hex string from a register updated daily by the electricity company.

One example from the original hex string would be : 463ACC01
This gets translated to a binary string : 01000110001110101100110000000001

This is the template used :

{% set STGE = states("sensor.zlinky_status_register_2") %}
{# Convert byte string into binary string #}
{% set STGE_binary = "{:b}".format(STGE[0]|int(base=16)).zfill(4) +
"{:b}".format(STGE[1]|int(base=16)).zfill(4) +
"{:b}".format(STGE[2]|int(base=16)).zfill(4) +
"{:b}".format(STGE[3]|int(base=16)).zfill(4) +
"{:b}".format(STGE[4]|int(base=16)).zfill(4) +
"{:b}".format(STGE[5]|int(base=16)).zfill(4) +
"{:b}".format(STGE[6]|int(base=16)).zfill(4) +
"{:b}".format(STGE[7]|int(base=16)).zfill(4) %}
{{ STGE_binary }}

This mostly works unless the leftmost bit is set to 1 (so hex string start with 8 to F).
In that case, the value is still populated in the template sensor, but HA throws an error in logs :

Error writing config for core.restore_state: Bad data at $.data[63].extra_data.native_value=10011011001110101101010000000001(<class 'int'>
Error writing config for core.restore_state: Bad data at $.data[63].extra_data.native_value=10011011001110101001000000000000(<class 'int'>

To investigate further, testing this in the Model editor in dev tools :

{% set STGE = "7FFFFFFF" %}
{# Convert byte string into binary string #}
{% set STGE_binary = "{:b}".format(STGE[0]|int(base=16)).zfill(4) +
"{:b}".format(STGE[1]|int(base=16)).zfill(4) +
"{:b}".format(STGE[2]|int(base=16)).zfill(4) +
"{:b}".format(STGE[3]|int(base=16)).zfill(4) +
"{:b}".format(STGE[4]|int(base=16)).zfill(4) +
"{:b}".format(STGE[5]|int(base=16)).zfill(4) +
"{:b}".format(STGE[6]|int(base=16)).zfill(4) +
"{:b}".format(STGE[7]|int(base=16)).zfill(4) %}
{{ STGE_binary }}

This reports a result.

Then if I change 7FFFFFFF to 8FFFFFFF for STGE no result and this error gets throwns in HA logs :

ERROR (MainThread) [homeassistant.components.websocket_api.messages] Unable to serialize to JSON. Bad data found at $.event.result=10001111111111111111111111111111(<class 'int'>

If I split this code into 2 sensors with first and second half of the binary string, it works fine without error.

What am I missing ?

[Sourcer63 (Johannes)]

@Narb

What am I missing ?

Int represents a signed 32bit number. When your most significant bit changes to 1, your result is negative and doesn’t match your hex anymore.

You could convert your signed to an unsigned expression. Not sure if this works, becasue the number range is still only half of what you want.

{{ [STGE_binary, 0] | max }}

[Narb]

Makes sense, thanks.
The proposed code throws an error though :

TypeError: '>' not supported between instances of 'int' and 'str'

[Sourcer63 (Johannes)]

Right, STGE is a string, not a number, hence the error.

But thinking about it, the int number range remains and you can’t convert a hex higher than 7FFFFFFF this way. So two sensors is a good work-around.

What exactly do you need the binary representation for?

[checking12 (Sander)]

Just a thought: wouldnt it be ‘easier’ to craft a python script to do this which you call on updated values? python function for this is really simple and powerful

[Sourcer63 (Johannes)]

I’ve found something:

{% set hex_str = "F63ACC01" %}  # Example HEX string
          {% set num = hex_str | int(base=16) %}  # Convert HEX → integer
          {{ '{:064b}'.format(num) }}            # Format as 64-bit binary

Result is
0000000000000000000000000000000011110110001110101100110000000001

[Narb]

@Sourcer63

What exactly do you need the binary representation for?

Each bit, or couple bits, is codified to have a specific meaning.
Like relay open or closed, type of contract, last error, many things like that.

[Sourcer63 (Johannes)]

I see.

My above template can also produce 32bit binaries:

{% set hex_str = "8FFFFFFF" %}  # Example HEX string
          {% set num = hex_str | int(base=16) %}  # Convert HEX → integer
          {{ '{:032b}'.format(num) }}

10001111111111111111111111111111

[Narb]

Sounds great, and shorter.
The 64bit version works, but the 32bit one behaves the same as my original code, error if leftmost bit is 1.

[Narb]

I can make it work with the 64 bits version, I just have to modify my resulting template sensors and add 32 to the position in the string :

{% if states("sensor.stge_interpretation")[24]|int(base=2) == 0 %}
  {{ "No" }}
{% else %}
  {{ "Yes" }}
{% endif %}

becomes :

{% if states("sensor.stge_interpretation")[56]|int(base=2) == 0 %}
  {{ "No" }}
{% else %}
  {{ "Yes" }}
{% endif %}

Thanks for your insights

[Troon]

You can use:

{{ hex|int(base=16)|bitwise_and(2**bit)|bool }}

to get the boolean state of the bitth bit of hex.

Pop this in the template editor to play:

{% set bit = 27 %} {# between 0 and 31 #}
{% set hex = "DEADBEEF" %} {# between 0 and FFFFFFFF #}
hex : {{ hex }}
bit : {{ bit }}
{{ "{:0=32b}".format(hex|int(base=16)) }}
{{ " "*(31-bit) }}^
bit {{ bit }}: {{ hex|int(base=16)|bitwise_and(2**bit)|bool }}

[justone]

Since here’s lots of template-editor geeks around …

What I don’t get (and I think to remember it was diffent in the past) is:

if I add

{% set hex_str = "F63ACC01" %}
{% set num = hex_str | int(base=16) %}
{{ '{:032b}'.format(num) }}

I get no output at all, eg. a blank result, but if I simply add something senseless such as in line 1 I get both the result from line1 aswell as a binary

{% set hex_str = "F63ACC01" %} {{ hex_str }}
{% set num = hex_str | int(base=16) %}
{{ '{:032b}'.format(num) }}

anyone with an idea about why this happens? Or am I wrong and it was like this in the past? I stumbled across similar things recently when I wanted to do some quick tests in the template editor while getting no results at all until I added some extra stuff.
Looks to me as if something’s broken but I don’t know where to look for that issue.

[Troon]

@justone

anyone with an idea about why this happens?

It’s the same error as the OP was having. The template editor tries to help by interpreting numeric-looking strings as numbers. This:

{{ "1110110001110101100" }}

returns what you’d expect, interpreted as a number, but this:

{{ "11101100011101011001" }}

returns 11101100011101012000. Add another digit and you get a blank result and an error in the system log:

Unable to serialize to JSON. Bad data found at $.event.result=111011000111010110011000(<class 'int'>

You’ve overflowed the signed 32-bit int limit. If you turn down your hex number to below 80000000, it works fine. If you put anything else in the output, it doesn’t try to help and the problem goes away.

[Sourcer63 (Johannes)]

@justone

I get no output at all, eg. a blank result,

Oh wow, indeed! Actually adding a # is sufficient to trigger an output.

@justone

Looks to me as if something’s broken

Possibly. Suggest that you open a separate thread with this finding or add as issue on Github.

[justone]

@Troon

and an error in the system log

Oh dear, I’d say it’ll be much nicer if such an error would be shown directly in the template-editor windows similar to warning similat to

{% set a = b %}
{{ a }}

where you get directly informed about ‘b’ is undefined.
Would easy things a lot without switching between template-editor and system log.

Since not being that familar with github any idea where to exactly file such a request?

[Troon]

@justone

Since not being that familar with github any idea where to exactly file such a request?

Heads-up: Feature requests are moving!

Hey everyone We’ve opened a new spot for feature requests: home-assistant · Discussions · GitHub The current Feature Requests section here on the forums will be locked soon. No new topics or replies will be possible. Over time, feature requests have shown up in all kinds of places, including this forum. But let’s be honest: this section has turned into one big, mostly unorganized bucket. There’s no structure, almost no labels, no categorization. Only search can help, and…

[petro]

We can only raise specific errors. Some of them are caught upstream and displayed as errors in the logs. It’s unlikely that the core team would allow these intercepted errors to be raised to the frontend because it would have unintended changes in things like template entities or frontend cards.

[justone]

No problem, it happened to me quite often that a {{ something }} simply resulted in no output at all. Perhaps it’s not really essential to know the exact error which might get detailed upsteam but sort of hint that the content should have produced something while the result remained blank.
I’m not into coding so I can’t say if something like that would be possible, but I think about newbies trying the template-editor for the very first time getting no result at all. It’ll be a once-tried and skipped experience perhaps and the template-editor is a damn usefull thing in fact.