I found what caused the error in my setup, you need to enable the home graph api. Previously in the documentation that was explained under the api_key instructions. Now it is described under the service account under point 9 and 10, this was changed 20 hours ago. This took me a few hours to understand, hopefully it will help someone else!
Direct link to activate the home graph api: https://console.cloud.google.com/apis/api/homegraph.googleapis.com/overview