You might be able to do this with an iOS Shortcut that runs a Home Assistant script that adjusts the TV volume ewhen the Phone app is opened and and another when closed (Shortcuts > Automations > New > App > Choose the Phone App).
edit: Just tried this, and it doesn’t work like you’d expect.
I also just found this thread that explains the problem