Ok, solved this. Just if others run into this:
ssh into hass with PI user
Install python3 dev:
sudo apt-get install python3-dev
Activate venv:
sudo su -s /bin/bash hass
source /srv/hass/hass_venv/bin/activate
Then install the missing dependencies with pip 3:
pip3 install pychromecast==0.8.1
Still have no idea why I had to install python3 dev again but it worked