I’m a newbie to this stuff, but I sensed when getting help on something a little similar, that this is the level/complexity where advanced users kind of recommend moving from a ESPHome config solution to a custom component solution in C++.
You could look over this and the GitHub project if you’re interested.
I would head over to Discord and see what the guru’s reckon.