Location aware voice assistant with Asterisk server not working as I hoped

Is it possible for HA to be aware of the actual phone being used to call the voice assistant when going through an Asterisk server? I have an endpoint set up in the Asterisk server so I can dial 5000 and have it call HA, but that makes it look like the call is coming from the Asterisk server rather than the phone. When I was looking at the SIP conversation while debugging issues in the early days of the voice assistant it appeared the address for where the call originated from was being included in the messages so it seems like that should be possible. Has anyone else run into a similar situation?