433toMQTTto433 Bidirectional Arduino Gateway

ha-username · August 29, 2019, 7:08pm

I have a few questions, if you or someone else wants to answer:

How to best integrate PiLight into HA? I could think that a demultiplexer could work - one which parses the JSON and auto-generates switches/sensor - but perhaps there are better ways.

–

How to tell the OMG PIO project to the ESPilight library from the lib folder (synced to a local git) instead of the one from github? I’ve changed the library include to be ./lib/ESPiLight but sometimes it is copied to my .pio/libdeps/nodemcuv2-test folder, sometimes the GitHub version is used.
Even with that approach, if I could get it working, I still have to manually delete the libdeps/…/ESPiLight subfolder before a build so that my library changes are recognized.
The other option is to modify directly into the subfolder and risk the chance that it might get overwritten with the GitHub version by an update/clean.

1technophile · August 29, 2019, 7:16pm

I will let HA specialist answer on this one.

Maybe there is remaining links, did you try to clean your .pio folder, here is some information that can help:

ha-username · September 4, 2019, 1:42pm

I’ve started working last night on the Nexus sensor integration but it proved to be quite challenging. According to some documentation (I did not do my own measurements) the sensors send 12 bursts of 36 bits i.e. 12 x 36 x 2 = 864 pulses. I haven’t seen any PiLight example that is able to deal with so many pulses and catch an incomplete train of them. Within so many pulses there are bound to be errors, typically spurious shorter pulses within the normal ones (e.g. 100us interspersed with the usual 500, 1000 and 2000us).
Essentially, one must be able to capture at least 2 trains, preferably 3 and see that they all carry the same data. This would not be so hard if there was a consistent way to detect the start and end of a train.

I will leave this aside for now until I can gather some more programming tips, it was not just a simple matter of adjusting timings.

1technophile · September 4, 2019, 6:59pm

Thanks for the feedback!

ha-username · September 28, 2019, 7:46pm

Just returned from vacation and had time to play a bit some more. The decoding was quickly done but the biggest issue was that ESPiLight had a minimum delay of 5100ms between messages. Changing this to 3700 allowed the nexus messages to be properly detected, as the length of the sync/repeat pulse is ~4000ms.
After cleaning up of the old code (I created a processor for a train instead of a single sequence) I will try to create a pull request for ESPiLight.

Question: currently all messages get published together under the home/OpenMQTTGateway_ESP8266_RF/PilighttoMQTT topic, regardless of the device being used. So I have both:

{"message":{"id":157,"channel":0,"battery":0,"temperature":22.6,"humidity":60},"protocol":"nexus","length":"157","repeats":2,"status":2}

and

{"message":{"id":16,"systemcode":341,"unit":7,"state":"off"},"protocol":"daycom","length":"16","repeats":4,"status":2}

under the same topic.

I’ve enabled Zmqttdiscovery but it doesn’t seem to do anything useful for the PiLight integration. I’ve tried adding a custom section at the end of ZgatewayPilight.pilightCallback() to try autodiscovery but I didn’t research exactly how it’s supposed to work.
My idea was to check if msg[“temperature”] is available in the resulting JSON and create a few discovery topics (similar to DHT). Otherwise, if there is a “state” message member, it’s probably a switch/binary sensor.

I don’t get it how the OMG + PiLight is supposed to integrate with HA. The only way I see it, an MQTT demultiplexer should be written which handles this: MQTT - topic - help
Is there any easier way?

1technophile · September 29, 2019, 8:08am

Depending on the code modified you should better submit it in the pilight repository directly.

Yes, mqtt discovery for pilight is not implemented. And currently we don’t how what would be the best way to do it.

There was some discussions here :

Maybe @ahmadk or @Petrica have some ideas.

ha-username · September 29, 2019, 12:17pm

Thanks for the answer!

Ugh, I realized this as I was cleaning the code. I think I copied the files for the protocol directly into the ESPilight repo folder, so my changes are now untracked. Need to figure out how to best do this under Windows, as it’s supposed to work with links. My C/C++ + VS skills are not that strong, I would rather let someone else handle all this and do a pre-review.

I started it but it proved to be a bit of a can of worms:

ESPiLight doesn’t send the device type or name back to the callback
Just adding the hardware type to the callback breaks all previous code
Default parameters cannot be implemented in a class, but it could be done by adding another callback
the protocol structure is not public so it cannot be returned - was thinking of opening up findProtocol()
while trying to do auto-detection back into OMG, I got unexpected resets a few times

It would be probably easier to do this “heavy” development on an ESP32 instead, as one can use OpenOCD for ICD. AFAIK this cannot be done with NodeMCU.

I started attacking this topic and managed to get it working in 1-2h. It’s easier to “develop” in YAML and do a quick automation reload. It’s ugly and has hardcoded bits but it does the job fine for now:

- alias: OMG Pilight demultiplexer
  id: 1a9ac126b34641c09824f583c33fd9f5
  trigger:
    platform: mqtt
    topic: home/OpenMQTTGateway_ESP8266_RF/PilighttoMQTT
  condition:
    - condition: template
      value_template: '{% if trigger.payload_json[''message''][''temperature''] %} true {% endif %}'
  action:

    - service: mqtt.publish
      data_template:
        topic: '{{ ''home/sensor/PiLight'' + trigger.payload_json.protocol|string + trigger.payload_json.message.id|string + ''/state'' }}'
        payload: '{{trigger.payload}}'

    - service: mqtt.publish
      data_template:
        topic: '{{ ''homeassistant/sensor/PiLight'' + trigger.payload_json.protocol|string + trigger.payload_json.message.id|string + ''_temperature/config'' }}'
        payload: >
          {"name": "PiLight {{trigger.payload_json.protocol|string}} Sensor Id:{{trigger.payload_json.message.id|string}} channel {{trigger.payload_json.message.channel|string}} Temperature","device_class": "temperature","state_topic": "home/sensor/PiLight{{trigger.payload_json.protocol|string}}{{trigger.payload_json.message.id|string}}/state","unit_of_measurement": "Â°C","value_template": "{{"{{ value_json['message']['temperature'] }}"}}","expire_after": 300}

    - service: mqtt.publish
      data_template:
        topic: '{{ ''homeassistant/sensor/PiLight'' + trigger.payload_json.protocol|string + trigger.payload_json.message.id|string + ''_humidity/config'' }}'
        payload: >
          {"name": "PiLight {{trigger.payload_json.protocol|string}} Sensor Id:{{trigger.payload_json.message.id|string}} channel {{trigger.payload_json.message.channel|string}} Humidity","device_class": "humidity","state_topic": "home/sensor/PiLight{{trigger.payload_json.protocol|string}}{{trigger.payload_json.message.id|string}}/state","unit_of_measurement": "%","value_template": "{{"{{ value_json['message']['humidity'] }}"}}","expire_after": 300}

The automations.yaml snippet above catches everything that’s published by OMG and creates separate state topics for each temperature sensor. It then publishes two new auto-discovery topics, for temperature and humidity, pointing back to the state topic:

Third topic: I ordered some time ago a relatively 433MHz PIR sensor from Amazon and thought it would be detected by PiLight. As it seems, it’s not, so I will probably have another protocol on the way. I’ve already decoded the transmission type (with URC) and just need to figure out how channel selection works. Would like to know if there’s anyone willing to ffer help for a code review before a pull request.

AhmadK · September 29, 2019, 4:36pm

As signals from all devices come to one gateway and it does not separate them into individual mqtt topics, a demultiplexer is needed if you don’t want to duplicate a chunk of YAML every time you add a new device.
I personally use a simple automation and a python script to do the job for DIGOO R8S sensors:

- alias: pilight2sensor demultiplexer
  initial_state: true
  trigger:
    - platform: mqtt
      topic: !secret pilight_bridge_state_topic
  condition:
    condition: template
    value_template: "{{ trigger.payload_json.protocol == 'tfa'}}"
  action:
    - service: python_script.set_values_from_pilight_message
      data_template:
        id: "{{ trigger.payload_json.message.id }}"
        temperature: "{{ trigger.payload_json.message.temperature }}"
        humidity: "{{ trigger.payload_json.message.humidity }}"
        battery: "{{ trigger.payload_json.message.battery }}"

but your use case might be different and require another approach. I do not use any autodicsovery features at all, don’t see any point so far, at least in OMG case (are there any btw?)

I’m in the same boat with Sonoff PIR2 - works fine with RF module, but is unknown to Pilight
Happy to do something to get it supported but don’t know where exactly to start…

Do you mean that ESPilight PR? I have some general C++/C background, but not sure it would help in such a specific case…
Anyway, I think you’re safe to make a PR and if something is not quite right, they’ll let you know

Petrica · September 30, 2019, 3:22pm

I use several 433 Mhz PIR and open sensors with OMG RF on Arduino Mega and ethernet shield, then a demultiplexer in HA. The OMG RF is, by far, the most stable smart device I have (the initial setup was done a long time ago and everything is manually setup in HA). Although I tried OMG PiLight, I found multiple decoding errors or two different codes for some of the devices successfully identified by OMG RF so I will stick with it.

For instance, the following two posts occur simultaneously when a PIR sensor triggers (from what I’ve found ev1527 is a generic protocol used by many 433 Mhz devices; also, I don’t own any Kerui D026 sensor).

home/OpenMQTTGatewayPilightTest/PilighttoMQTT
{"message":{"unitcode":34499,"state":"opened"},"protocol":"kerui_D026","length":"","repeats":2,"status":2}
Creating RF PiLight buffer
Pub json into:
home/OpenMQTTGatewayPilightTest/PilighttoMQTT
{"message":{"unitcode":507038,"state":"closed"},"protocol":"ev1527","length":"","repeats":2,"status":2}
Creating RF PiLight buffer

For environment sensors (temperature, humidity and light) I use different solutions:

ESP8266 with DHT22&LDR with OMG code for all rooms except bathroom (due to high moisture I didn’t wanted to go anywhere near the mains); each sensor posts the readings in its own topic so no need for a demultiplexer;
Digoo R8H weather sensors with RFLink (currently not supported by PiLight) for bathroom and inside the fridge For the bathroom I use another R8H with the humidity sensor de-soldered and substituted with a LDR (humidity scale is 0 to 100% which was much easier to work with compared to the temperature scale published by the temperature sensor).

ha-username · September 30, 2019, 3:40pm

Cool setup.

You say that PiLight gives errors but I see that you pasted an OMG Pilight trace.
I’ve initially thought that RF would be the way to go but PiLight actively supports a lot more protocols and is just not worth trying to implement each from scratch. Personally, I’ve found that rtl_433 supports everything I own, but I had issues with undervoltage, CPU usage and randomly dying decoder process.

How do you power them? The biggest advantage, for me, for these R8H sensors is that they are self-contained and batteries last for >1 year. One of them has been sitting outside in the rain for months now and still works.

Why LDR? I thought humidity is a more practical application for a bathroom.

Does the demultiplexer look similar to what I’ve posted above and the one by @AhmadK ?

Petrica · September 30, 2019, 4:25pm

Yeap, one of the issues I have with PiLight concerns decoding: for one PIR sensor is decoding two payloads, while for another PIR (of the same brand) is only showing only one.

I use standard USB phone chargers (nothing fancy, however I try to avoid knockouts). I literally have one such device in every room except for the bathroom.

However, for the RF gateway I use an Asus powerbank as it supports passthrough charging and cost is lower than of an UPS.

My R8H batteries are only good for 7-8 months (I tried with both rechargeable and regular batteries).

There was a misunderstanding : I use two R8H units in the bathroom (one untouched that provides temperature and humidity and one modified that provides light intensity readings). Although payloads from R8H is received at each 30 seconds which I consider sufficient, the LDR sensors in the other rooms send the payloads based on changes in the light intensity immediately so that the automation for the lights can resend the payload if bulbs didn’t react.

Pretty much similar, except in my case the RF output is not in json format so it only deals with simple payloads.

denisj · October 17, 2020, 11:03am

Hi there Petrica,
can you tell me how do you receive data from DIGOO DG R8H please ?
Do you use the OMG Pilight ? with witch hardware ?

Thanks a lot
Multumesc mult

Petrica · October 17, 2020, 12:57pm

Hi Denis,

Currently I’m using RF Link (Arduino Mega) to receive data from R8H (most of the time it is pretty much stable but I would like to move all 433Mhz receiving part to an independent device). I know that Florian is working to add support for it in the OMG, however I haven’t been able to closely follow the topic. I will then update my oldest and most reliable smart device, which is a 4 years running Arduino Mega with one of the very early 433Mhz OMG builds.

Although R8H are pretty nice devices and very cheap, I’m more fond of an all-in-one solution consisting of NodeMCU + DHT22 + LDR with OMG for running sensor nodes in each room. With passthrough powerbanks, these devices run for months (one of them even for 27 months) without any reboot being needed whatsoever.

I mostly use R8H in environments with high humidity, where I don’t want to fiddle with electric related dangers, such as in the bathroom and the fridge After replacing the humidity sensor with a LDR (I could have replaced the temperature sensor but the humidity uses a 0-100% scale more easy to work with, rather that dealing with both positive and negative values), drilling the case and using lots of hot glue I also ended up with a reliable light intensity sensor; it ain’t gonna win a beauty prize but it is perfect to be used with motion sensor in low light conditions (I could have placed LDR directly on the PIR motion sensor in order to trigger it only in low light but it would limit PIR usage with an alarm system).

You’re welcome! (Cu placere )

denisj · October 18, 2020, 10:40am

Dear Petrica, thanks a lot for your answer.
This can be a solution, but cause I’m so new in this world, can you tell me about powerbanks,
in order to make it work for about one year please ? how much batts (18650?) need it ?
Or if you have a link that explain me the combination of NodeMCU + DHT22 + LDR with OMG please.

Here is the link where Florian tell us that DIGOO R8H has been added to Pilight:

But I don’t understand so well… maybe we must whait some type of compilation or release of the Pilight

Thanks a lot again
Denis

Petrica · October 18, 2020, 3:43pm

Hi,

I think I might need to clarify what my previous note means:

It’s not that the NodeMCU with all the sensors can run for 27 months of a single powerbank charge (regardless of powerbank capacity). It means that the powerbank can prevent any situation where a reboot of the NodeMCU device is needed (either due to an error within the code or due to a mains power failure). The powerbank runs in passthrough mode, thus the 5V needed for the board is still fed by the mains, though a USB charger.

In order to be able to run a NodeMCU for a long period on a battery then a sketch with deep sleep enabled functionality would be needed, which is beside the point made by me above as the NodeMCU would sleep for most of the time and only wake up and send data under certain conditions which are time related (ie. wake up once per hour/day and send sensors data to MQTT) or triggered by an external module (this is the functionality of the battery operated Tuya devices which employ a secondary MCU that will wake up the ESP82xx when the sensors are triggered, then ESP82xx connects to wifi and sends its payload; however, this process takes several seconds and it is not really suitable for cases where immediate feedback is needed). It should also be noted that wifi is quite power hungry and cannot be expected to run for long periods of small batteries if continuous connection with the smart home software is needed (not even 18650s could provide power for extended periods).

In respect of powerbank brand, they should specifically include passthrough mode in their description in order to be able to both charge the internal accumulator and provide power to external devices (I use 10,000 mAh Asus devices and I know Xiaomi has several in their portfolio). There are 18650 modules on the Chinese sites that claim similar functionality but it is always difficult to find a reputable seller who did not cut corners.

Here you can find the necessary documentation for building an OMG with Platformio (it’s not that difficult, at least not when compared to the Arduino build, where you didn’t easily knew what software library version was needed for which project )

https://docs.openmqttgateway.com/upload/pio.html

My personal use case is for the data from the LDR sensor (light intensity data) to be sent to HA immediately after a certain delta from previous value is registered (I use this data in order to adjust motion triggering lighting, so, even if the PIR sensor triggers, the bulbs only turn on if there is insufficient light). For the humidity and intensity data I only need them once per 5 minutes and I use these data for monitoring the environment condition in my house (to turn on heat - with a relay connected to a NodeMCU - or AC - with IR blaster and IR receiver connected to NodeMCU). As humidity and temperature they don’t change very often it this is ok for me; however, there might be cases where much lower time delays would be needed (I helped someone build a greenhouse monitoring system consisting of about 40 NodeMCUs with DHT22 sensors sending data on a 10 seconds interval).

If you only started going the smart home route I suggest reading a lot of the documentation in multiple topics. It is quite a steep learning curve; however, once things start moving on, I think that time invested is totally worth it.

I think that only Florian can advise on the status of the implementation.

denisj · October 18, 2020, 5:35pm

Thanks a lot Petrica.
I just read in some place that the R8H units can be at max 3 pieces in same place.
Cause everione can be set from 1 to 3 … is that true please ?

I think that the best solution can be to make from zero an arduino
with temp/humidity sensor and RF module.
All this in order to make it work with 433toMQTTto433 or the Pilight
Put all in sleep mode and wake it up only 5 minutes.
In this case with 2 batts of 1,5V maybe will work one year.

Thanks again
Denis

Petrica · October 18, 2020, 5:55pm

The sensor has a switch with 3 positions to choose from 3 channels. Actually I think that this refers to the fact that, from the factory, they have the ability to set a single ID to a batch of devices, then each device will provide its unique key by changing the channel case there are two units with the same ID.

I don’t know anything about their production process in order to compute probability for more than 3 devices in the same batch sent to customers to share the same ID, but I guess it would be quite low.

Haven’t looked too much into a 433toMQTTto433 option but I think there might be solutions such as Zigbee that are optimized for low power, even lower data transmission (but permanently connected with gateway), that could offer years of a LR2450 battery.

denisj · October 18, 2020, 6:12pm

The problema is that I don’t know why… but I don’t like the Zigbee
Maybe cause in moy house I allready habe BT + WiFi + RF433 … a lot of radio.
So I don’t want to add also Zigbee.

In the mean time that Florian will apply the R8H ,
and because I have a lot of time this mouths cause I work from home (corona)
I will search and study some projects that can be usefull in my case.

What I want is to put a temp/humidity sensor in all my rooms and also outside
in my greenhouse and in the chicken coop… and all this sensor data I want to have it
in the Hassio (Node-Red) in order to make some rutines.

So because I so that you are a guru in this thinks I ask you permission to vrite you privatelly
and maybe when you have some minutes we can talk about at phone (I’m in Italy)

Thanks a lot again
Denis

Petrica · October 19, 2020, 7:36am

Hi,

I’m not familiar with Node-Red as I only use yaml automations (since I don’t work in IT, I had no previous programming experience to compare with writing yaml automations), but once the sensor data is in, then I guess there would not be any issue with NR.

Yes, you can contact me.

1technophile · October 19, 2020, 4:50pm

Hi,

Yes we are waiting for the release of Pilight new version, and when released it could be integrated to ESPilight and OMG