I wanted to share with anyone who is using regular Asterisk and PJSIP that I was finally able to get calling to HA working. Here is what I had to do:
Install opus codec - basically follow the directions from Asterisk Tutorials: How to install and use Opus codec in Astersik
Add pjsip endpoint to pjsip.conf
[5000]
type=endpoint
context=users
disallow=all
allow=opus
callerid="Home Assistant"
I think the endpoint name (5000) can be whatever you want, it will get used in the next step.
Create an entry for dialing HA in the dialplan extensions.conf
exten => 5000,1,Dial(PJSIP/5000/sip:ha@<HA IP>:5060)
Replace with the IP address homeassistant is running on.
After that I was able to dial 5000 from my office phone and tell homeassistant to do stuff. Maybe this is obvious to people that have been using Asterisk for awhile but it took me a long time to realize I needed the PJSIP endpoint.