I am looking for the same solution and as per my finding the issue is that Modbus integration in HA is focused on serial communication. RS485 uses RE (Receive enable) and DE (Data enable) pins which are not covered in serial implementation in HA.
So the solution would be to implement a Full duplex communication like was suggested above.
In this way you would need 4 TTL-RS485 adapters. Two of them in the master (HA) with Raspberry TX pin connected to DI (data input) for the first converter, and RX pin to RO (Receive output) of th second converter. Also the same in every slave.
I am testing it right now so I cannot confirm it works.
What I know is that ArduinoModbus library covers this scenario and enable/disable the pins RE/DE while sending/receving because it is designed for RS485.
HA Modbus integration is designed for serial TTL not for RS485.
By the way, I believe that if you use the USB-RS485 adapter it will work with HA Modbus integration because you don’t need to enable/disable the RE/DE bits.