Tips welcome, ESP for ESPHome with serial Interface to Arduino MEGA for Pins

ESPHome
1

Dear Forum,

as stated in my introduction post, I have build RoomClients that are serving one to three rooms. These read out some analog values and digital sensors, do PWM to control lights and read digital inputs.
All this is controlled by an Arduino Mega with an Ethernet shield and MQTT. Since I am struggeling with the Ethernet/MQTT part of the system for a long time, I would like to take a different approach.

Recently I got myself one of these:

An Mega with Mega pinning for compatibility with my boards and an ESP8266 for ESPHome, which is awesome.
But here is the catch. The Arduino port expander ESPHome lib does not work for me, since I need those digital sensors and PWM as well as the new board connects both controllers via RS232 instead of I2C.
I have recently compiled some self made code for an ESPHome sensor, so I am familiar with inserting self made code into an ESPHome configuration.

So much for the introduction. Here is my question:

  • Does anyone have a good tip on a Serial communication protol library that I can use? I need something reliable, easy to understand and code.

  • With the last project I did to get my self made code into ESPHome, I was not able to create more than one type of sensors from that code. So I have a lot of “Sensors” which give floats, but I was not able to create additional “binary sensors”. My workaround was to create more float sensors that now represents 1/0. I am guessing the error is infront of the keyboard, but maybe someone has a hint :wink:

That’s all folks,
Christian

2

What is the problem you are having with Ethernet and MQTT? Adding an ESP8266, serial communication between the two and WiFi seems to be adding layers of unreliability compared to Ethernet.

3

Run appdaemon and you’ll be able to write one in python for back and forth serial.

my system is down right now but from memory you can use the run every API to keep a function continuously scanning. i.e:

def initialize(self):
        start_time = self.datetime() + datetime.timedelta(seconds=1)
        self.handle = self.run_every(self.you_function_to_be_called, start_time, 1)
    def you_function_to_be_called(self, kwargs):
        ser = serial.Serial(
            port='/dev/ttyACM0',
            baudrate=57600,
            timeout=1,
            parity=serial.PARITY_NONE,
            bytesize=serial.EIGHTBITS,
            stopbits=serial.STOPBITS_ONE,
        )
do soem cool stuff that you want to send....
   if ser.isOpen():
            ser.write(data)
            ser.flush() 
            try:
                doc = ser.readline()
            except Exception as e:
                self.log("Error : Incomplete Msg")
                pass
        ser.close()

do some stuff with the doc reading
4

if your using mqtt you can just set that up through the home assistant side and pass entities through to app daemon. this is your friend
https://appdaemon.readthedocs.io/en/latest/AD_API_REFERENCE.html

Side note I agree with @AaronCake seems like your adding more room for issues by adding a third protocol in between which could be achieved via the other two

5

So my issue with the existing setup is that I get random connection losses. The only thing that helps is resetting the Arduinos and Ethernet Shields. Then It reconnects to the DHCP and MQTT Server and is up and running again.
With my limited debugging possibilities, I was not able to track down the problem. I checked the Ethernet and MQTT examples a lot of times, making sure there is no user error. I have a feeling that the Ethernet shield is not made for 24/7 connectability. But that`s just me pointing fingers.

You might be right, I might just exchange one problem set with another. But: I have roughly 20 ESP’s with ESPHome running happily in my home automation without any major trouble. So I do trust that the ESPHome Ethernet and HomeAssistantCommunicationProtocol side is build by more and a lot better programmers than I am.
Therefore I am expecting a lot less problems on that side. I am fairly good at C/C++ but I never worked with python. Therefore I would like to stick to ESPHome. I have programmed a lot on microcontrollers and low level protocols to be able to get the RS232 part stable and/or debug it.

So my goto is as mentioned having ESPHome do the communication and WLan stuff while my code is implemented like this:

I have done this for another small project and it runs fluently.

Cheers,
Christian

6

Maybe to do with this :

1 Like
7

I was just about to suggest this, RPI can have an annoying habit of resetting the port, causing your Arduino to reset every time it opens communication.

8

How about an ESP32 with an ethernet port. I’m not sure you’d even need an Aurduino then?

9

I do have this behaviour on half of my boards. Therefore, cycling the power does not work and I must reset those manually. I will install the hack, but this will not solve the random communication loss, it will just solving it a bit easier since the 24V lines for the room clients have a separate distribution panel in the basement.

10

I think this is another issue. I would guess that the RPi also somehow tests the DTR (?) line, which is normally used to reset the arduino. Therefore you get the reset.

11

I thought about this a while ago. at least the ESP32 replaces the RoomClient Shield thing. But, it does not have enough pins. I need roughly 14 PWM outputs, 14 Digital INs, and 4 sets of sensor inputs for a DHT22, a light sensor and a LEDdriver output.
Surely I could just but port expanders to the ESP32 but this would also complicate things. IN addition to that, I would have to design a new shield. This would present no problem but cost time and money and wasting the perfectly good shields I have right now.

The ESP32/Arduino Mega Board would just be plugged into my RoomClient Shield and be good to go.

12

If you are getting connection issues then it likely is an issue with the Ethernet shield you are using, or a code problem. If you post your Arduino code it might be possible to spot the problem.

There are a lot of cheap Ethernet shields out there and lots of them have issues. The reset issue posted above is common (though I’ve never experienced it). Ideally the reset line for the Wiznet chip would be removed from the reset button and connected to an Arduino pin so you can reset the Wiznet problematically. Depending on the shield, this can be accomplished simply by removing the pushbutton and running a jumper wire from the appropriate pad to one of the Arduino pins.

Another common issue are incorrect resistors used between the connection of the PHY to the RJ45. The appropriate value is 49.9 ohms, but a lot of clones use 500 ohms! This has the effect of causing the shield to only be able to negotiate link with certain devices, or not at all, or only after multiple resets, etc. Swap the resistors with the correct value or connect another resistor in parallel:

Another great option is to swap to the EtherMega or EtherTen by Freetronics.

Yeah, the price is higher, but everything is integrated into one board, it is well designed, the designer can actually be contacted. And what’s a few more dollars for rock solid reliability?

13

Interesting. 510 instead of 49.9 Ohm… I will check my shields. Although I have to hit a very good day of eyesight and trebling to change the 0402’s :scream:

I’ll digest the code down to its Ethernet/Mqtt parts and post it later.

So why does nobody trust ESPHome with a wireless connection? From my experience this is nicely stable…

14

I don’t think it is a ‘nobody trusts wireless’, but more of a ‘wireless is not as reliable’ thing. Esphome is quite reliable from my experiences, but the fact will always remain that my wifi devices always have more down time than ethernet. I think of how many times I have ssh’d in to my router over ethernet, to fix some issue with wifi. All these times, ethernet devices never new there was a problem. Doesn’t happen often, but the fact that this happens makes wireless at least one tick less reliable than ethernet. That isn’t even talking about interference issues that can pop up at times… what if installing a new high tech ceiling fan (or other noisy addition) creates enough local rfi to kill the wifi to an esp? That won’t ever happen with ethenet.

Bottom line, yes esphome is great for lots of things, but given an easy alternative of using ethernet (wires already there or easily installed), then that is the preferable tech to use. Really it is more like ‘use ethernet over wifi if at all possible’.

15

In short, wires are generally more reliable, but you are clearly having the opposite experience. This points to a hardware problem.

16

I just spotted this https://www.olimex.com/Products/IoT/ESP32/ESP32-EVB/open-source-hardware

17

ok, let’s say I’d go for Ethernet. Then two possible routes are open:

Fixing the Arduino Ethernet

I’ll show the code today…

Getting an ESP32 + Ethernet up and running

@nickrout: The board look great, but it would still have too little I/O since there is no port expansion in this PCBA:

esp32pinning
esp32pinning848×584 169 KB

So the route here would look like this:

  • Get the ESP8266 with RS232 to Arduino communcation up and running (sinceI already have this one). Yes, first implementation with WLan
  • port the ESPHome code to ESP32 and Ethernet (should be done in minutes, since its just reconfiguring the ESPHome file)
  • create an easy “shield” pcb to plug into the Arduino to get an reliable system
  • done.

This comes now back to my initial question: Anyone knows a nice RS232 communication protocol that I can just use?

Cheers,
Christian

18

My humble opinion is that wireless is a great convenience, but anything that doesn’t require mobility or i snot temporary should be wired.

Again, my opinion, but when I think of permanent home automation devices I am thinking 70 years into the future (I figure by that time I’m dead and won’t care). Will WiFi be around in 15 years? Probably, but will 802.11b be now deprecated? The WiFi Consortium has always maintained backward compatibility but at the current rate of evolution, that has to change at some point. When 802.11whatever is announced in 2032 or whenever and breaks 802.11b/802.11g, it would sure be fun replacing all those light switches, outlets, etc.

Ethernet ain’t going anywhere. 100 years from now, Ethernet devices will still be able to negotiate 10MB half duplex.

19

It can be also a functionality to add to OpenMQTTGateway as we already support Arduino MEGA (if you fix your ethernet shield issue).
If you are interested I can help you with that. Currently, it supports reading one GPIO only and controlling one but it can be easily extended by some code.

For the anecdote, I had an Arduino UNO + an ethernet shield running OMG for several years and this was one of my most reliable home automation components.

20

I did a brief lock on the website and I am confused.

Here is what I think I understood: So for an ESP you have binaries, for my Arduino I would take a new project, include the OPENMQTTGateway and my cleaned up code. What would be the benefit of using OPENMQTT against a self programmed MQTT communication (If it is obvious, please excuse my ignorance :wink:)