Have you considered Zanzito and MQTT as your framework? I had your same concerns about Google Home and Alexa so I ended up sprinkling cheapy wall mounted tablets and connected speakers around the house and having Zanzito with always listening voice control running in the background. Building up the skills is a bit more time intensive, but I used this thread as the starting point and have very few problems since.