Integrating Home Assistant with custom python Voice Assistant

Development
1

Hello!

I’m currently planning my future smart home and some python AI behind it.
Currently I’m building a smart voice assistant using python and chatterbot, it can already perform tasks like:

  • Full AIMP music control, including playing songs by name and translating song names
  • Telling time and date
  • Math calculations
  • Checking social network messages and reading them if asked
  • performing book-search on multiple sites by book’s name, and building a list of available links and downloading closest match if the book is available for free
  • using multiple voice-recognition services for best results and fail-safe (google, bing, yandex and sphinx)
  • groundwork for future command-processing
  • Chatting with a self-learning chat bot
  • Shutting PC down and rebooting
  • Checking PC status

And I have a lot more stuff in mind.

So, thats basically a back-end right now, and I’m thinking about how to make it’s front end and integration with Home Assistant. Usually I go for PySide\PyQT GUI, but I think that for this task it would be better to have an app running on the server, and accessible from any device from inside the local network and outside of it too. Like just picking up your phone, openning it’s url and talking to it, sending data to the server, processing, responding, etc.

Sending commands to HA seems pretty straight forward,using it’s Python API.

So, I’d like to hear any thoughts about that. But I have one question right away:

Is it possible (and how if it is) to use HA’s voice recording\recognition to send commands to some other python app running on the same machine or something? What would be the easiest way to do it?

Basically, I don’t want to dig into HA development, at all if possible, or at least not much, just somehow building my ‘AI’ around HA. Since I basically only need to have ability to input voice commands using multiple devices through web interface, I thought that using HA’s already working UI seems logical.

If not, I will dig into web-ui development, to build my own web ui for my Voice Assistant, while only using HA to access smart-home features (lights, sensors, etc).

Thank you for any help and advice.
Cheers.

1 Like
2

I might be able to help… Do you have skype?

3

Hi, sure, it’s Nixellion.

But I solved most of the problems so far :slight_smile: I actually made my assistant a flask-server with JS front end UI. Works like a charm. Still did not yet even setup hass, but i imagine using it’s api should be straight forward.

4

@Michael_Davydov This sound really great! Is your project public yet?

5

No, I keep it very private, and learning a lot of stuff along the way. I also hard code a lot of sensitive information into it right now, like some passwords and logins. But once I’m happy with most of the stuff, and once I actually bind it to HA, I may put it on git. There is also an issue - I’m making it for Russian language only yet, but I am trying to put multi-lang support as well.

So far it runs on my DIY Ubuntu NAS machine, it works as a voice-controlled bot through Google Chrome UI, and is also accessible as a Skype chat-bot (well, just made an account for it, not ACTUAL skype-bot).

So far it can control plex media server (OpenPHT player, music mostly, but the ground work for other stuff is already there), turn on\off ALL lights through HASS (since I only have 3 hue bulbs to play with yet, will get more stuff around march-april), tell me jokes about Chuck Norris, tell some interesting quotes from books, find books by title and show them in UI with DL links, google stuff, show recipes, share files through Skype, translate text, and chat using python chatterbot by gunthercox.

And adding a skill is as simple as putting a new py file into ‘skills’ folder, subclassing the base skill class, and setting your activation phrase mask (using regex) and writing your logic.

But it’s all a real mess right now, a lot of stuff to clean up. Hopefull I’ll get there some day :slight_smile:

6

Hello, your project seems really interesting, i am actually planning to do something like this in the thai language. It would be a great help if you could recommend on how it works? I have been researching regarding this topic.

7

How did you manage to access the mic in Home assistant and send it to your python voice assistant?

Thanks in advanced.

8

Hello,

It’s the other way around. Voice Assistant is it’s own standalone python program\script that can run on any device within your network and it just ‘talks’ to HomeAssistant through it’s REST API.

So for example you can have HASS running on a your Main Server, and voice assistant is running on a Raspberry Pi Zero W with a mic “hat”. Pi Zero records audio, processes it into text, recognizes command and then just sends “turn this light on” to Home Assistant.

This is one way of doing it anyway. Another more proper way would be to have “server” and “sattelites” approach, which is similar to how Google Home and Alexa and others approach it. Meaning that you would run your Voice Assistant Server alongside HomeAssistant in a central “powerful” server. And you will have weak Rasperry Pi Zero’s or even ESP devices with microphones scattered around your home. these are called “satellites”. These Sats would be only responsible for recognizing hotword and activating audio recording. They would then send recorded audio to the central server and it will handle all the processing - speech recognition, NLP, intent recognition, etc.

You don’t have to write your own, however, because what I just described is pretty much how Rhasspy works. It’s come a long way since I worked on my own assistant.

Also I’ve abandoned my voice assistant and the idea of voice assistants in general because it’s more of a toy than a useful feature - for me and my family anyway. We stick to automations, physical buttons and controllers like Aqara Magic Cube or whatnot, and phones.

But we also live in a relatively small flat, not a house, so there’s only so much ‘smart’ we can stuff in here