Taking Home Assistant to the next level. Home Assistant.. Assistant? - AI-powered Machine Learning for HA

Development
1

Hey guys. So, I’m a programmer / AI (Machine Learning) Dev… and I was talking to a few other devs ( @ThisITGuy @dgomes ) about a pretty killer idea for a Home Assistant service

It would be a feature that used Artificial Intelligence / Machine Learning to suggest automations. For example it might say “hey we noticed that every day at 9:00 AM you ring your Tile Item Finder… would you like to set up an automation?” It could also do more complex stuff for example: “we noticed that whenever there’s motion downstairs on a Tuesday, Wednesday or Thursday, when it’s cloudy you turn on the pantry lights… would you like to set up an automation?”

It would be basically like the Nest Learning Thermostat but for the entire home.

But it would probably be a project where i’d need other devs on board though

A few devs I spoke to said they were interested, but the more the merrier.

How could this be done? Well you would just need to feed the Algorithm (I’m thinking a simple SVM or even Decision Tree) the data and it would make the predictions based off that.

Where would we get the data? Well fortunately Home Assistant already logs all of this.

image
image2880×1800 525 KB

This could be used as Training Data.

This would obviously have to be done securely given Home Assisant Users preference for privacy. An algorithm like this can be done locally. Now you’re saying “yea okay… many HA users run the server on a Raspberry Pi and similiar devices… you can’t AI on those little devices.” And 10 months ago, you may have been right. But recently Google released Tensorflow Lite, which is specifically designed for microcontrollers. A raspberry Pi would be significantly overpowered for what TFL needs.

image
image2432×852 280 KB

Tensorflow Lite is designed for microcontrollers. It runs beautifully on a RPi.

As @dgomes suggested, we may be able to use the Persistent Notification System to suggest automations. Or it could just be in it’s add on and the user could check every so often.

And of course it would have to ask permission before adding an automation and not be able to do anything on it’s own. I don’t know about you but I certainly don’t want some “Alexa, Open the pod bay door!” situation. Lol, all jokes aside… I don’t want Tensorflow deciding when it wants to open or close my freakin August Smart Locks. So it would need to suggest automations.

Anyway, I’d love to hear your thoughts

57 Likes
2

@robmarkcole I bet this will interest you :slight_smile:

3

I’am also interested for this project.

2 Likes
4

Exciting project.

2 Likes
5

Love the idea! I can see so many posibillities. To bad I’m not a programmer…

2 Likes
6

It would also need to ignore everything already controlled by automations
Sounds like an interesting prospect. How do you see it working? An addon?

1 Like
7

Or docker?

8

Sounds interesting. Maybe it could integrate with scheduler-component or clippy.js?

9

@chrisvacc would be happy to assist. Having worked on several ML type applications for HA I have a few comments:

  1. Going for a full on/official integration would be a lot of work. Going the custom integration allows faster iteration on ideas. What you are suggesting could be a hassio addon. Alternatively using mqtt statestream you could create a solution that is agnostic to the actual platform it is deployed on.
  2. Simple setup is very important for user adoption - this does mean the dev often needs to make many assumptions for users and this is a tricky balancing act. Start simple and add features over time as users suggest ideas
  3. Not all users are on RPi, and a solution that works on RPI might not translate easily to other platforms. Tensorflow installation is a good example. If you can use a ‘pure python’ approach that may pay dividends in support hours saved
  4. It should not be a black box solution - you will get many ‘why does it do this’ type questions, where what people want is some insight so they can tune the solution

An MVP might be:

  • Monitors logs/events
  • Highlights repeat daily events to users (lots of parameters to optimise here!)
  • Suggests an automation - could even just be some yaml the user then pastes over
2 Likes
10

And…

  • Recommends correlations on seemingly random events

Things like “I noticed that every time this wall switch is toggled, that camera stream in the same room got interrupted.”

5 Likes
11

This sounds like a really cool idea. I don’t have much to add from a development perspective, but I would certainly be interested in testing it out. I spend quite a lot of time working on what automations make sense and how to implement them. This could really put HA over the top regarding the simple utilitarian automations that home automation is all about(imo.)

2 Likes
12

This kind of things was the reason we added a context ID to all events and state changes. It allows to figure out what events relate to one another. See also the docs about context.

If you install the Jupyter community add-on, you will find the example Jupyter Notebook hooked up to your instance that uses context to determine your most popular entities that you interact with per time of day. See also the quick start docs.

Taking this next level to TF would be awesome :ok_hand:

8 Likes
13

this is exactly what I want a true smart home to be able to do!

How can I be of help?
I am also a developer, do python and would be glad to build up some ML knowledge.

I can offer a rather big playground for testing, although its kind of busy being operational. :wink:
But I have a lot of MQTT and Hue stuff, that could also be hooked to a dev system.

However, kid requires a lot of attention, ongoing phd thesis and part time work of course…

but I want want want this!

2 Likes
14

This is exactly what I would like to have. Context sounds like a correlation between events because they are already coupled in some way in the virtual world. This could actually find correlations between events that are not coupled in the virtual world yet, only in the real world.

I am a developer (mainly c#) but have never done anything with python or AI/machine learning, would be interesting in doing something with it once in a while.

15

What about the chronological scope of an event?
X happens and Y hours later happens Z. They are correlated, not by this period of time, but by some other event/condition.
How far back should events be correlated?

16

I’ve been doing something similar 2 years ago, I’ve been working on a python app that uses a [Solcast API] (http://solcast.com.au/) to predict the solar photovoltaic production and tells you ‘hey you might turn on your dishwasher between 10:30 and 12:00 to be synchronized with your solar production’…

it looks like this (it looks ugly I know, bad at UI :confused: ) :

image
image1122×676 43.3 KB

This might be useful given that home assistant already stores the power profile of each home appliance.

9 Likes
17

Hi, I’d love to be a part of the development process here. This is exactly what I think future smart homes need to do. I have a basic python understanding and have done several projects with it. However, ML is new for me but I am surely eager to learn. I might not be the best fit for the software part but I have a lot of installations running on home assistant, which we can definitely use for quality control and extensive user testing.

If you want we can get in touch with all the devs on board and help out where we can!

1 Like
18

I bought this book last week. Maybe it is also interesting for others:
TinyML: Machine Learning with TensorFlow on Arduino, and Ultra-Low Power Micro-Controllers

3 Likes
19

This looks like an interesting project!

I’ve also had my eyes on https://docs.ambianic.ai/ for a while now. But it looks like the shifted more towards Image recognition.

20

I though about it a little.

I think an unsupervised learning method like K-means clustering would make sense for an initial classification stage of all the available data in HA.

What do you think?