Mashine Leaning (ML) in Hass

1

Considering the huge amount of data available for Hass for most of the home I think it would make sense to hook up ML to Hass, but I am not sure if this is within the projects scope of should be a separated project.

The basic idea is that ML consume all the data from Hass, and categorize them to human inputs, and conditions, and the aim is to either suggest automation rules, or just replicate the human inputs automatically.

The goal is do less manual inputs, and button pushes, and voice commands, just let the machine do it, of course apart from the ML challenge there is the data challenge first, while most of the time all kind of sensors are available, the history isn’t long enough to make patterns out of it, but this can be solved easy, or sometimes a specific sensor data is missing, this is a harder to crack as you don’t know what you don’t know.
For example you have the temp in one room but not in an other, or you have external conditions like wfh day vs bank holiday etc, but I still think that the concept would work and it would be beneficial overall for the project to have the “hands off just let it learn” mode and after some time, certain actions will not be needed from the end-user itself.

The same way nest learns your heating demands, and the same way you just let the car wipe the windows and turn on the beamer lights.

Thoughts ?

2
1 Like
3

Love this video. I’ve referred to it no less than a hundred times in the last three months.

For you (tl;Dr) folks skip to the last four minutes of the vid…

1 Like
4

Hey @tom_l that is an interesting experience but not really relevant to my suggestion, LLM’s and other conversational / generative AI is obviously by the nature of it will make mistakes and misunderstand things.
My suggestion is purely behavior replication. You as a user switch the light on by night when it’s already dark in the house and you are home, this is something which can be detected by a machine learning algorithm, for example:
The ML come up with an automation suggestion, based on observation that when value X, Y, Z looks like this, then event A, B is triggered by the users, then it would put this suggestion into a dry run / soft lunch phase, so when next time value X, Y, Z looks the same, wait for event A and B and you see it, then you get a score that the automation is good, if not then you get a negative score, after some time everything which reached above a set threshold, can be either presented to the user, or depending on risk appetite just put into action and deploy it to live.
You can even add supervised learning in with making suggestion to the ML which values to take into consideration when making a new version of an automation which is almost perfect but not 100%, exceptions like bank holidays, or school breaks or having guests, or a specific guest etc.
Fundamentally you can say that it’s just Hass automation, as you can and should be able to all of it with Hass automation or Node-red if you prefer, my suggestion is that writing a good automation is hard, and ML can help with it.

Reason why I considered tight integration with Hass ( maybe even part of the project ) is that I believe that even human written automatons can benefit from it, for example, if automation misfires, and turn on the TV when it shouldn’t have, and the ML detect that right after the automation took an action the human undid it, then it can review the differences in conditions and make a suggestion what else to add to the existing automation to make it better, like you have a guest in the living room sleeping on the sofa, and the light turns on when they they switch sides because you forgot to add it as a conditions or turn of the specific automation, etc etc

5

It’s totally relevant.

What you want is not yet possible in any reliable way.

It is the ultimate goal though and will be possible eventually.

6

Care to extend on how you come to the conclusion ?

I am having a struggle to make the jump, we have a very clear case for a combination of supervised and reinforcement learning, we have historical data, and we have the bases for a good reward function, and while it would be super slow to learn, I don’t see what is the missing link which makes you say that it would be unreliable.
And just to be clear I myself also prefer XAI / XML wherever possible, and I think that would be the best way for Hass too, specifically the ML would produce Hass Automation suggestions rather then taking actions in your behalf, and that would already guarantee that it would not start hallucinate, and etc.

7

I never tried it but:

8

By reading about current attempts.

What you are asking for is currently science fiction.

9

UserX, I’m totally with you on this. What you are describing not only does make sense, it is a logical evolution of HA. Nest Thermostat tried to do something like that probably a decade ago - use current time, weather forecast, interior temperature, target temperature, and built-in motion sensor readings to train an ML model so that it could learn to set the right temperature at the right time on it’s own. It had limited success probably because of very limited inputs/data.

If I may share some thoughts…

Are user actions learnable in principle? ML can only learn something what has patterns and correlations. It can’t learn user turning lights on an off every hour based on flipping a coin. Or guess that a user has a fever today and wants different thermostat setting than usual. But it could learn that user always turns on the living room lights a minute after the car beacon gets back in range while it’s dark outside. Or that user dims kitchen lights to 10% every time after turning them on between 2 am and 5 am. Or that user turns AC off after opening 2 or more windows.

A thought experiment: could you ask somebody to observe you using HA for a month and then ask that person to control your devices the way you would, without talking with you? If the answer is yes, chances are it’s possible to train an ML model to do the same - but have to make sure ML model gets the same inputs (e.g. espresense detecting person’s apple watch moving from one room to another may be an equivalent of seeing the person walking).

What model to chose? My expertise in this is very limited, but I suppose it would need to be one of the time series forecasting models where every second (?) you’d give it entire state of the system (every value of every entity/attribute), and get the prediction of the entire state for the upcoming second. The controllable entities (e.g. value of the light switch) could be used as triggers if they differ from the previous second. Probably it would take a lot of fine-tuning and filtering. Maybe actual trigger should be enacted only after e.g. 10 consecutive predictions that the light should be on.

I tried something like that years ago by feeding values of everything I could get out of my SmartThings into a recurrent neural network. I never completed the project because of how massive the datasets turned out to be and the lack of user friendly ML platforms for time series forecasting back then.

Today there are lots turn-key models on AWS, Azure, and elsewhere. This may be worth another try.

One part which I find very confusing is the feedback loop. The model like that would inevitably make mistakes and will need continuous refinement. How would that work? Simply turning off the light which model turned on undesirably would just create a wrong correlation - the model would learn that the light was needed very briefly. Maybe the only way is to go back and retrain with the corrected value, but that would be very expensive/cumbersome.

It would be interesting to hear from ML experts.

The “GPT Took Over My Home” video seems to be about the challenges of using GPT to translate vague natural language requests into HA commands. I didn’t spot anything about training a ML model HA user behavior in order to replicate it.