Taking Home Assistant to the next level. Home Assistant.. Assistant? - AI-powered Machine Learning for HA

Development
61

As an ML/DL engineer, my first reaction to this thread has been “why not brainstorm a number of potential use cases?”. So, some possibilities (with those already mentioned labeled as Existing);

  1. (Existing) Have an AI monitor periodic actions, and then suggest an automation to fit a recognized pattern. Bonus: Have the AI prepare the automation, and present for concurrence/correction.

  2. (Existing) Have an AI analyze the solar forecast and recommend scheduled allocation of dispatchable loads (e.g., dishwasher, laundry, ev charging, etc).

  3. (Extension of 2.) Seek the lowest overall carbon emissions by training an ML to predict the low CO2 emissions times in one’s area by training on past CO2 Signal hourly data, then combining with solar PV generation (if present) to recommend scheduled allocation of dispatchable loads, and adjustment to one’s thermostat setting (e.g., if very high CO2 emissions between 3-5pm on a hot day, then raise thermostat set point by 2 degrees F). Another example might be instead of charging one’s EV at 1-4 PM during a very high CO2 emissions level, allow the net solar generation to offset the high emissions, and charge overnight during a very low emissions time. There could also be another version of the model to analyze the aforesaid with high tariff time-of-use and/or real time pricing to include cost as a consideration.

  4. When no resident is present, trigger a sequence of actions to turn on and off lights in a manner representative of the typical usage pattern by day of week, month, sunrise/sunset, etc, (perhaps with some randomization).

  5. Have an AI learn when and by how much a dimmer is typically lowered per a particular pattern (e.g., dinner in the dining room around 7pm, etc)

  6. Notify when a sensor-monitored window is open when a cool/cold front is moving in. Could even learn the Thermal Time Constant of the home and ask if there is an non-sensored window open if the temperature in the house or room drops more rapidly than expected when a cool/cold front moves in.

  7. An NLP could be trained to recognized an innocuous but specially coded sequence of words to alert law enforcement or set off an outdoor siren if a thug or armed robber broke into one’s house and was holding you at bay.

  8. Because both solar forecasting services available in HA are so inaccurate, could create an ML model framework so that anyone could train a model with one’s own local weather forecast data and solar output results. I’m seriously considering this, and have undertaken similar efforts with fairly good success. My only concern is the cloud data may not be precise enough to do better than ForecastSolar and SolCast.

I’m delighted there is an JupyterLab integration, though I have not tried it out yet in the HA context. I’m running HA on an RPi 4, so any training/validation/testing on that would have to be in low CPU priority mode, and likely take many days if not weeks. A better option may be to collect the data on HA, then transfer it to one’s home computer, or whatever ML/DL environment they have (with Kaggle as one free option).

Two very important interrelated areas in AI are interpretability and explainability (with the terms often used interchangeably). Predictions, classifications, recommendations, etc often leave the user puzzled about how the AI arrived at such an answer. There are a large number of open source tools and methods available, indeed so many that a number of attempts at taxonomies have been undertaken, with this one a kind of Rosetta Stone.

image

3 Likes
62

There are a lot of different approaches to realize predictive automation even with ai. Ultimately the highest insecurity is the human itself. Even if all parameters, which lead to a certain situation, are the same, the behavior of people can or will differ due to emotional circumstances or whatever. And in addition: as long as it is nearly impossible to track which human is in which room and why, it will, ai available or not, nearly be impossible, to realize really helpful predictive automation. And let alexa ask i.e “The situation is as follows, would you like to switch the lights on?” will lead to annoying increase of communication. Currently, my wife and me will be informed about the dryer and washer, some reminders, and some states which have been reached or missed. Only that is sometimes kind of annoying regardless that these information help us making our life more comfortable.
I was more euphoric a year ago starting to thinking about some predivtive behavior of HA. I stopped my activities due to the issues described above.
There are some key functions missing, before we can push that again.
Just my 2 cents

63

@Pirol62 So what you appear to be saying is that there may be a number of instances when use cases #1 and #5 may end with the user saying “No thank you”, though there may be another number where they say “Why yes, make that automation”. And then there are still the other 6 use cases above that could be valid for those whose situations apply. And if you believe that HA AI is not for you at this time, no one would begrudge you that stance.

For others who are interested, in addition to AI Interpretability and Explainability noted previously, AI-Human Interaction is an intertwined field that is also growing in leaps and bounds. One dimension is level of iinteraction, where Gartner posits the following 7 levels;

Level 1 — Human Only: Normal Decision Making without AI

The person seeking to make a decision views available data, perhaps including simple analytics, to determine for themselves the potential courses of action to take and makes the decision themselves. This is the standard course of decision making without the use of AI.

Level 2 — Advisory: Avoid Catastrophic Human Mistake

The person still creates the decision path, though presents this to the AI for feedback and potential reconsideration. An example might be a radiologist who examples a CT scan for signs of early lung cancer. After deciding that the scan does not show signs of cancer, the radiologist engages the AI to review the scan in a manner similar to a second opinion or the advice of an internist. If the AI indicates probable cancer signs, the radiologist would then re-examine the scan in the area identified by the AI.

Level 3 — Recommendation: Limited-Context Machine Decisions

The AI analyzes the data to identify one or more courses of action, scoring each with supporting explanations, and presenting them all to the person. The person reviews the recommendations and explanations, then makes the final decision choice

Level 4 — Approval: Broad-Context Machine Decisions

The AI analyzes the data to identify one or more courses of action, scoring each with supporting explanations, and selects one course of action to take. The person reviews the recommendation and explanations (perhaps choosing to review the others), then approves the recommendation or takes some other course of action.

Level 5 — Veto: Avoid the Human Bottleneck

This is the inverse of the previous step, where now the person may be only monitoring the AI decisions, though choosing to intervene when in only the instances where the person disagrees with the course of action the AI is about to undertake.

Level 6 — Audit: Trust, but Verify

The AI identifies one or more courses of action, determines what it considers the optimum one, and takes the course of action (which could be null, as in no action). On a periodic basis (e.g., daily, weekly, monthly, etc), a person audits and scores the set of decisions. If at any point, the metrics of the decisions undertaken fall below an acceptable level (or are on a trajectory to), a retraining or structural update to the model and/or explainability solution is considered or automatically triggered.

Level 7 — Autonomous: Exception-Based Review

This is the same as Level 6, except that a review only takes place when an exceptional condition or outcome occurs.

Regardless of the complexity of the system or the number of levels one has plans to advance to, there is always a gray area as control shifts back and forth between the human and the AI. Yet, focusing on the human centered design during the entire build and all stages of the AI ensures more trust is garnered throughout the entire experience. End users, as well as those that deploy AI should be included from the beginning of the design to increase understanding, AI reliability, overall buy-in and trust.

1 Like
64

I haven’t went through the whole thread but has anyone got anything off the ground with this yet?

I’ve had some moderate success training a basic dense neural network to predict certain actions based on the sensor states but looking to see what other approaches are. I’m looking to incorporate various models into the system such as computer vision and NLP but the dense network has been just are starting approach for me so far.

From what my experience so far they is a fair bit of data engineering i’ve had to do, in regards to getting things where i want them. Just curious to see what other approaches have been so far and what everyone’s aims are?

Long term i’d like to develop a model capable of reading energy inputs from various devices and effectively take various actions in order to save money on energy costs while not disturbing user comfort etc.

2 Likes
65

Hello! I want to join the topic, but as a test because I don’t have much knowledge. If I may.

66

I think someone is already working on something very similar to this idea - take a look at The Silly Home addon

67

It’s a pity there aren’t enough videos about this integration. And it’s unclear how it works. ))))

2 Likes
68

MetaAI has released an Llama model which is much smaller than ChatGpt and standford created a refined model called Alpaca using relatively little resources.

69

I tried SillyHome briefly, but it doesn’t really do much at the moment and I am not seeing much in terms of meaningsful development.

70

I’m a developer. And im also in for this project. It*s time to add som intelligence to home assistant.

1 Like
71

I’m for using AI to assist in programming the Automations, Scripts, Blueprints, Configurations, Cards, etc.
This is and excellent Project!

1 Like
72

I would like to use ML/AI for optimizing my under floor heating with heat pump. Had an conversation with ChatGPT about it and in the end let it to write a spec:

Objective:
Develop an AI/Machine Learning model to optimize the operation of a house heat pump, taking into consideration the house’s thermal inertia, fluctuating electricity prices, and desired temperature range.

Primary Goals:

1.	Determine the house’s thermal inertia, indicating how rapidly the house cools or heats in relation to external temperatures.
2.	Maintain the house’s internal temperature within a specified range.
3.	Minimize energy costs by leveraging varying electricity prices, including potential overheating during cheaper electricity hours to benefit from the house’s thermal inertia during more expensive hours.
4.	Generate an hourly heating plan for a day in advance based on forecasts.

Data Available:

1.	Room Temperature Data:
•	  Current room temperature.
•	  Historical room temperature data.
2.	Temperature Preferences:
•	  Desired room temperature: 22°C.
•	  Maximum allowed temperature: 23°C.
•	  Minimum allowed temperature: 21°C.
3.	External Temperature Data:
•	  Current outside temperature.
•	  Forecasted outside temperature (hour-by-hour for the next day).
4.	Electricity Pricing Data:
•	  Hour-by-hour electricity prices from Nordpool for the current day and the next day.

Steps for Implementation:

1.	Data Collection & Preprocessing:
•	Gather and clean the data sources mentioned above.
•	Convert all data into a uniform time series format (hourly readings, ideally).
2.	Exploratory Data Analysis (EDA):
•	Analyze historical room temperature data in relation to external temperatures and heating activities to understand heating patterns and the house’s inherent thermal characteristics.
•	Study the correlation between external temperature, heating activity, and room temperature to preliminarily understand the house’s thermal inertia.
3.	Model Development:
•	Use regression or time series forecasting models to predict the house’s temperature response to external factors (outside temperature and heating activity).
•	Integrate electricity pricing data to develop an optimization algorithm that can decide when to heat and by how much, with the objective of minimizing costs while maintaining the desired temperature range.
•	Utilize historical data to train the model and validate its predictions.
4.	Heating Plan Generation:
•	Using the optimized model, forecast the room temperature for the next day, hour-by-hour, based on the forecasted outside temperature and expected electricity prices.
•	Generate a heating plan that suggests when to turn the heat pump on or off and potential heating intensities if variable control is possible.
5.	Evaluation and Iteration:
•	Compare the model’s predictions with actual outcomes.
•	Refine and retrain the model as more data becomes available and as seasonal patterns emerge.
5 Likes
73

That sounds very interesting. Now the real “Magic” would be creating the actual addon/automation/blueprint to make it happen…

74

Im also experimenting with AI ! Got a blocker … and I hope community can help.
Logs from my HA a very limited fud to the limited number of devices … 3 lamps and 3 switches :frowning:
It’ll be great if anybody can share example of a more complex log from 3-5 days …

75

Love the idea, gots some spare hardware floating around to start playing, just no idea where to start haha

76

I’m wondering how this can be integrated with the new voice assistants.

77

It’s coming at some point.