ES032 with MCP23017 and a relay board

Hi, I have an ESP32 connected with a MCP23017 and use MCP230xx I/O Expander — ESPHome to overcome the small pin amount in my ESP32 (D1 Mini Esp32 Esp-32 Wifi+bluetooth Internet Of Things Development Board Based Esp8266 Fully Functional 10pcs - Integrated Circuits - AliExpress).

I have connected multiple 5V board relays (4x4). All components are fed with a 5V stabilised PSU. The relay board themselves are powered separately.
I do not use “external” resistors.

All relays controlled by the MCP23017.

A while back all was broken and it turned out the MCP23017 died, replaced it and all worked again.

Now I have another issue but I cannot get grip on the cause and I would like some help…
Some other sensors are connected to the ESP32 which allways keep working during the above issue. Also not alle relays boards “loose functionality” at the same time.
It seems that one relay board (not always the same) stops working. Sometimes remotely restarting the esp32 resolves the issue and makes all responsive again.

I can keep using the switches in HA and on the html page in esphome but they just physically do not respond anymore.

Completely power cycle the entire setup always restores all functionality untill it breaks again…
Untill I am gone from home and I cannot do anything physically…

The setup worked for about 8 month without any problem. Then after replacing the MCP23017 it worked agian for about 2-3 month without any issue and now since 2-3 months I have the above issue.

Any help/thought is highly appreciated.

Thank you…

No reply, so hard question probably…

I happened again (4th time).

Now I have some extra behavior information. It seems that my MCP23017 “breaks” when logic switch inputs and outputs are switched very quickly (to relay boards) it seems it is broken after that.

Which brings me to the a question I cannot really find a good answer to:
Do I need output pull up/down resistors between the (virtual) GPIO ports from the MCP23017 to the relay boards?
Currently I have them directly connected which functionally works great. Relay boards are powered separately via 5V.