Taking Home Assistant to the next level. Home Assistant.. Assistant? - AI-powered Machine Learning for HA

This is a great idea @chrisvacc.

My use case is, to predict the amount of energy, that can be produced ny by solar system today, based on the weather situation (sunny, cloud, Rain) and the remaining time until sunset. I need this amount of energy (in kWh) to decide, weather to turn on the charger of the electrical car, or weather to postpone it to later (if possible). I have changed the use case from the one, that @dadaloop82 has provided, perhaps it helps to see another one.

My thoughts on this, after reading the posts carefully:

1. Prediction of sensor states

The prediction of sensor states, based on other sensor states is a very good way to start this. I would not reduce it only to switches. The sensor state of a switch is “on” or “off”. I could be possible to predict the state of sensors state with a probability of x percent.

2. Configuration

The configuration of such a component in the configuration.yaml could be like this:

machine-learning:
  models:
    - name: "Erwarteter Solarertrag heute"
      unique_id: expected_solar_output_today
      state_class: measurement
      device_class: energy
      unit_of_measurement: kWh
      start_prediction: 2021-05-01 00:00 +02:00
      add_future_values: true
      target_entity: sensor.solar_energy_produced
      input_entities:
        - entity: weather.myaddress
        - entity: sun.next_setting
      include_context: true
    - name: "my second model"
      unique_id: my_second_model
      ...

The context, explaind by @balloob, could be the way, to relate the states and the events; but that is not clear in my head right now.

3. Execution environment

The execution of such a prediction can be done by the execution environment of https://appdaemon.readthedocs.io/. Perhaps, we ask @ Andrew Cockburn, what he thinks about this concept. The execution can be fast or slow, depending on several external facts (CPU, time frame, amount of input_entities, ans so on); so we would expect the results, after the calculations are done (1 Minute, 10 Minutes, 2 hours). The questions ist, if we nee some feedback to get to know some information about the status of the calculations.

There will be the first execution of the prediction (the calculation of the first model). That would take a while, since the (recorded) history of all included entities since the start_prediction have to be calculated.

Then, the model has to be stored somewhere persistent, to stay alive also durings reboots.

After that, that, the existing model would be extended, when one of the input entities trigger a change: The model calculation would be triggered again, but now it will eventually run in shorter time, since only one change of an included entities has to be calculated. The updated model would have to be stored again, and the predecessor model can be overwritten.

4. Output

The output of such a ML prediction could be an sensor, based on the configuration above, and with attributes, e.g. like this:

entity: expected_solar_output_today
state: 2.1
attributes:
  - type: ml-sensor
  - probability: 79 %
  - state_class: measurement
  - device_class: energy
  - unit_of_measurement: kWh
  - friendly_name: "Erwarteter Solarertrag heute"
  - trace:
    algorithm: "The name of the ML algorithm"
    - calculations:
      first_prediction: 2021-08-19T19:57:27+01:00
      last_prediction: 2021-10-19T19:57:27+01:00
      number_of_predictions: 567
    ...:

If one of the input entities trigger a change, the model calculation itself would be triggered and the ml-sensor would be updated as described.

The trace adresses the very importan point, that @robmarkcole made: “why does it do this?” People will create further automation on the ml-sensor; and they need to know something about the quality of the sensor. It will be different, if the ml-sensor has 10 input values, or 1000. And it is important, from which time frame the values come.

5. Naming

The name of such a component could be machine-learning. This is, what is done here. It could be a custom component.

3 Likes

Hi @klacol!

I really thank you so much for the very valuable information you provided.

At the moment, due to force majeure, the project is temporarily shelved but I would like to resume it absolutely as soon as possible.
I should resolve some personal issues within a couple of weeks and then get back to work.
I’d love it if you’d join my Github channel to collaborate together, even if it’s just with ideas.

Thanks so much!
Daniel

Hello! The “Compensation” integration in HA needs to be more advanced. I would like a button to feed in data from entities, to do some faster (or longer) calibrating. Or maybe having more than pairs of numbers… Case and point: predicting power usage of a split heat pump so that it can closely match available solar power based on factors like interior, exterior temp, its delta, etc so that an automation will select the best scenario.
Or even more awesome, showing real time heatpump COP and finding out if is better to crank up the heat output now at high exterior temp (thus better COP) or leave as is, based in some weather forecast or even local temp falling rate.
Update: used https://stats.blue to get a model (formula) for COP. Templated power*cop formula and then Riemann integration on the result to get equivalent gas in kwh for a heat pump.

1 Like

@dadaloop82, this is really interesting.
I’ve started collecting data in a SQLite database with the idea to create an algorithm that could predict next actions.
I used pyscript (that is much easier for me) for handling the database.
In order to limit the number of records I created two automations that trigger when the status of my devices change, one for covers and one for lights.
I’m now changing the database, trying to make it more meaningful, with more columns(data): each record should be a picture of the status of all entities in all the house.
So, now the database is really stupid, but once decided how to set it up I could share with you in order to let you evaluate your system with different data.
I’m open to suggestions for the set-up of the database.

EDIT:
@chrisvacc , sorry I missed to tag you.

1 Like

@chrisvacc Hi, I see a lot of interest with the posts but very little response form your side. Is this still active, i.e. are you still working on this?

Good morning everyone, sorry for my long absence.

I’ve been thinking a lot about applying AI to Home Assistant and I’ve done many experiments in this regard, but unfortunately they never gave me satisfaction.

I tried to change my point of view and wanted to express my experiences in this thread I just wrote in this Forum. I would be very pleased to receive your comments and proposals, your thoughts and concerns, in order to arrive together to a possible solution, in a constructive way.

Thank you!

I am a data-scientist with a background in Computer vision. Currently want to play with time-series data for my next project. Also it would be interesting to do a ML project in IOT/Smart Home area.

There are several ideas for me to try, but I am not HA user and have only a week long data from my setup (about 40 sensors and devices)

If you can share your data set it would be a great start

I know what you mean. I just started thinking of building a system with machine learning.but making it truly learn something is terrible complicated. Letting HA suggest automation based on stuff you do every day is fun but not really smart. Because it will only know this if you do the exact same thing every day. In which it’s easier to just write an automation for it and be done with it.

In my opinion AI needs to have a data set and being able make the connections on its own. So if I come out of bed one hour later it still needs to know what to do. And when I don’t work at home on Wednesday this week but a Thursday it needs to recognize that and act on it based on its findings. So it’s needs to recognize patterns that can differ in 1000s of ways but still knows what to do. I guess if you really want to go this far it’s simply impossible to write with a few lines of code. Google is spending 1000nds of man-hours a week on this and they can’t even set my alarm clock to go off earlier when there is a traffic jam. It’s very complicated. You have 1000s of variables that need to be checked every second

1 Like

@sygys I so totally agree with you. I am following this topic out of curiosity. But I often shook my head reading the high hopes that were expressed here. I am convinced that a neural network will rarely be able to infere the deeper motivations of our actions from the patterns of the sensors and actuators in our houses. It would surely produce occasional hits, but they would be embedded in a large number of wrong actions. These fails would be frustrating to all users and would totally ruin the acceptance of the system. Sorry for disenchanting you guys, but IMHO this is not feasible in the magical way that you are envisioning it.

Hey Team! I’ve started this same project earlier this year. Is there a slack channel that can be joined to jump into and participate the discussion and work?

@Jpsy At the moment im trying to find a way to being able to correct the system and then making the automations that did the wrong actions idle. So far it not very efficient and requires allot of additional lines of code to the automations. At the moment when an automation triggers it will turn on an input_boolean start the device and 10 seconds later turns the boolean off. The moment a device changes state another automation checks if the boolean went on. If so the system now knows the device was turned on by an automation. if a user turns the device back off and the boolean is off. the system knows the user turned a device on or of manually and another boolean will turn on that indicates that this device was turned on manually and this boolean stays on for the period i choose. This allready makes life easier if you have for example a AC that is programmed to function full automatic but you dont want it to turn on at the moment. The system will know the AC turned of by the user and wont turn back on anymore until the next day.

But this approach is terrible complicated to do for every device. I would love to see some sort of attribute for every device in home assistant which stores the last action and if it was triggered by an automation or manually. The AC is a hard component in this because we have an IR remote that turns on the AC without HA knowing. So thats why i did this approach at the moment and it works like a charm.

1 Like

Could you share this code?

Not sure this is possible directly in HA. But I think you can do this in Node RED using HA API calls. I didn’t try but it should be possible to add custom attributes to HA entities through the API.

You can use ESPHome and its IR component to let HA listen to the IR commands of your handheld in parallel. So HA becomes aware of all changes issued through the remote.

this is an automation that triggers the AC. it turns on the boolean that will indicate it did go on with an automation:

- id: '1628602498e629'
  alias: Airco woonkamer verwarmen en koelen
  description: ''
  trigger:
  - platform: time_pattern
    minutes: /5
  - platform: state
    entity_id: input_number.gewenste_temperatuur_woonkamer
    for: '00:00:05'
  condition: 
  - condition: not
    conditions:
      - condition: state
        entity_id: sensor.room_temp_thermostat_otgw
        state: 
          - 'unavailable'
          - 'unknown'
          - '0'
      - condition: state
        entity_id: sensor.tuin_weerstation_temperature
        state: 
          - 'unavailable'
          - 'unknown'
  action:
  - choose:
##### optie aan koelen#####
    - conditions:
      - condition: state
        entity_id: input_select.woonkamer_verwarming_modus
        state: 'koelen'
      - condition: time
        after: '06:00:00'
        before: '23:00:00'
      - condition: state
        entity_id: input_select.woning_status
        state: thuis
      - "{{states('input_number.gewenste_temperatuur_woonkamer')|float < states('sensor.room_temp_thermostat_otgw')|float }}"
      - condition: state
        entity_id: climate.airco_woonkamer_esp
        state: 'off'
      - condition: state
        entity_id: binary_sensor.woonkamer_dubbele_deur_sensor
        state: 'off'
      - condition: state
        entity_id: binary_sensor.achterdeur_sensor
        state: 'off'
      - condition: state
        entity_id: input_boolean.airco_woonkamer_handmatig
        state: 'off'
      sequence:
      - service: input_boolean.turn_on
        target:
          entity_id: input_boolean.airco_woonkamer_aan_uit_gezet_met_automation
      - service: climate.set_temperature
        data_template:
          hvac_mode: cool
          temperature: 22
        target:
          entity_id: climate.airco_woonkamer_esp
      - delay: 10
      - service: input_boolean.turn_off
        target:
          entity_id: input_boolean.airco_woonkamer_aan_uit_gezet_met_automation
      - delay: 5
      - service: climate.set_fan_mode
        target:
          entity_id: climate.airco_woonkamer_esp
        data:
          fan_mode: '1'
      - service: notify.discord_webhook
        data:
          message: 'Airco woonkamer op koelen'
      - delay: 5
      - service: select.select_option
        target:
          entity_id: select.airco_woonkamer_vertical_swing_mode
        data_template:
          option: up
##### optie aan verwarmen#####
    - conditions:
      - condition: state
        entity_id: climate.airco_woonkamer_esp
        state: 'off'
      - condition: state
        entity_id: binary_sensor.woonkamer_dubbele_deur_sensor
        state: 'off'
      - condition: state
        entity_id: binary_sensor.achterdeur_sensor
        state: 'off'
      - condition: state
        entity_id: input_boolean.airco_woonkamer_handmatig
        state: 'off'
      - condition: state
        entity_id: input_select.woonkamer_verwarming_modus
        state: 
          - 'hybrid'
          - 'elektrisch'
      - "{{states('input_number.gewenste_temperatuur_woonkamer')|float > states('sensor.room_temp_thermostat_otgw')|float }}"
      sequence:
      - service: input_boolean.turn_on
        target:
          entity_id: input_boolean.airco_woonkamer_aan_uit_gezet_met_automation
      - service: climate.set_temperature
        data_template:
          hvac_mode: heat
          temperature: "{{states('input_number.gewenste_temperatuur_woonkamer')|float + 5}}"
        target:
          entity_id: climate.airco_woonkamer_esp
      - delay: 10
      - service: input_boolean.turn_off
        target:
          entity_id: input_boolean.airco_woonkamer_aan_uit_gezet_met_automation
      - delay: 5
      - service: climate.set_fan_mode
        target:
          entity_id: climate.airco_woonkamer_esp
        data:
          fan_mode: '3'
      - delay: 5
      - choose:
        - conditions:
          - condition: state
            entity_id: binary_sensor.eettafel_person_motion
            state: 'off'
            for: '00:05:00'
          sequence:
          - service: select.select_option
            target:
              entity_id: select.airco_woonkamer_vertical_swing_mode
            data_template:
              option: down
        - conditions:
          - condition: state
            entity_id: binary_sensor.eettafel_person_motion
            state: 'on'
          sequence:
          - service: select.select_option
            target:
              entity_id: select.airco_woonkamer_vertical_swing_mode
            data_template:
              option: up
  mode: single

Then you have this to check when a state changed to the AC if it was an automation or not:

- id: '165347030dewc0016'
  alias: Check of airco woonkamer handmatig aangezet is
  description: ''
  trigger:
  - platform: state
    entity_id:
    - climate.airco_woonkamer_esp
    to:
      - 'cool'
      - 'dry'
      - 'heat'
      - 'off'
      - 'heat_cool'
    from:
      - 'cool'
      - 'dry'
      - 'heat'
      - 'off'
      - 'heat_cool'
  condition:
  - condition: state
    entity_id: input_boolean.airco_woonkamer_aan_uit_gezet_met_automation
    state: 'off'
  action:
  - service: input_boolean.turn_on
    data: {}
    target:
      entity_id: input_boolean.airco_woonkamer_handmatig
  mode: single

This will turn on a boolean to set it to manual when it was a manual operation.

And last i have an automation that controls the AC once its on but only when not manually turned on:

- id: '162aanpasse4564gerfn8629'
  alias: Airco woonkamer aanpassen tijdens koelen
  description: ''
  trigger:
  - platform: time_pattern
    minutes: /3
  - platform: state
    entity_id: input_number.gewenste_temperatuur_woonkamer
    for: '00:00:05'
  condition: 
  - condition: state
    entity_id: climate.airco_woonkamer_esp
    state: 'cool'
  - condition: state
    entity_id: input_boolean.airco_woonkamer_handmatig
    state: 'off'
  - condition: state
    entity_id: input_select.woonkamer_verwarming_modus
    state: 'koelen'

  action:
  - choose:
##########################################
##### optie 1 temp lager dan gewenst #####
##########################################
    - conditions:
      - condition: state
        entity_id: input_select.woning_status
        state: thuis
      - condition: template
        value_template: >
          {% set gewenste_temp = states('input_number.gewenste_temperatuur_woonkamer') | float %}
          {% set binnen_temp = states('sensor.room_temp_thermostat_otgw') | float %}
          {% set verschil = gewenste_temp - binnen_temp %}
          {% if (verschil > 0) %}
            true
          {% endif %}
      sequence:
##### TEMPERATUUR #####
      - service: climate.set_temperature
        data_template:
          hvac_mode: cool
          temperature: >
            {% set temperatuur = states('input_number.gewenste_temperatuur_woonkamer')|float +1 %}
              {% if temperatuur > 30 %} {{ 30 }}
            {% else %} 
              {{states('input_number.gewenste_temperatuur_woonkamer')|float +1 }}
            {% endif %}
        target:
          entity_id: climate.airco_woonkamer_esp
##### DELAY #####
      - delay: 5
##### PRESETMODE #####
      - choose:
        - conditions:
          - condition: state
            entity_id: climate.airco_woonkamer_esp
            attribute: preset_mode
            state: 
              - Normal
              - Powerful
          sequence:
          - service: climate.set_preset_mode
            target:
              entity_id: climate.airco_woonkamer_esp
            data:
              preset_mode: Quiet
        default: []

###########################################
##### optie 2 temp gelijk aan gewenst #####
###########################################
    - conditions:
      - condition: state
        entity_id: input_select.woning_status
        state: thuis
      - condition: template
        value_template: >
          {% set gewenste_temp = states('input_number.gewenste_temperatuur_woonkamer') | float %}
          {% set binnen_temp = states('sensor.room_temp_thermostat_otgw') | float %}
          {% set verschil = gewenste_temp - binnen_temp %}
          {% if (verschil == 0) %}
            true
          {% endif %}
      sequence:
##### TEMPERATUUR #####
      - service: climate.set_temperature
        data_template:
          hvac_mode: cool
          temperature: >
            {% set temperatuur = states('input_number.gewenste_temperatuur_woonkamer')|float %}
              {% if temperatuur < 16 %} 
                {{ 16 }}
              {% elif temperatuur > 30 %} 
                {{ 30 }}
              {% else %} 
                {{states('input_number.gewenste_temperatuur_woonkamer')|float }}
              {% endif %}
        target:
          entity_id: climate.airco_woonkamer_esp
##### DELAY #####
      - delay: 5
##### PRESETMODE #####
      - choose:
        - conditions:
          - condition: state
            entity_id: climate.airco_woonkamer_esp
            attribute: preset_mode
            state: 
              - Normal
              - Powerful
          sequence:
          - service: climate.set_preset_mode
            target:
              entity_id: climate.airco_woonkamer_esp
            data:
              preset_mode: Quiet
        default: []

##################################################
##### optie 3 temp 0 tot 1 graad hoger dan gewenst #####
##################################################
    - conditions:
      - condition: state
        entity_id: input_select.woning_status
        state: thuis
      - condition: template
        value_template: >
          {% set gewenste_temp = states('input_number.gewenste_temperatuur_woonkamer') | float %}
          {% set binnen_temp = states('sensor.room_temp_thermostat_otgw') | float %}
          {% set verschil = gewenste_temp - binnen_temp %}
          {% if (verschil < 0 ) %}
            true
          {% endif %}
      sequence:
##### TEMPERATUUR #####
      - service: climate.set_temperature
        data_template:
          hvac_mode: cool
          temperature: >
            {% set temperatuur = states('input_number.gewenste_temperatuur_woonkamer')|float -4 %}
              {% if temperatuur < 16 %} {{ 16 }}
            {% else %} 
              {{states('input_number.gewenste_temperatuur_woonkamer')|float -4 }}
            {% endif %}
        target:
          entity_id: climate.airco_woonkamer_esp
##### DELAY #####
      - delay: 5
##### FANMODE #####
      - choose:
        - conditions:
          - condition: not
            conditions:
            - condition: state
              entity_id: climate.airco_woonkamer_esp
              attribute: fan_mode
              state: '1'
          sequence:
          - service: climate.set_fan_mode
            target:
              entity_id: climate.airco_woonkamer_esp
            data:
              fan_mode: '1'
        default: []
##### DELAY #####
      - delay: 5
##### PRESETMODE #####
      - choose:
        - conditions:
          - condition: state
            entity_id: climate.airco_woonkamer_esp
            attribute: preset_mode
            state: Quiet
          sequence:
          - service: climate.set_preset_mode
            target:
              entity_id: climate.airco_woonkamer_esp
            data:
              preset_mode: Normal
        default: []
    default: []

This is only 3 of the 15 automations i currently have for controlling our 2 Ac’s

1 Like

I got all of this. What I wanted to tell you is, that it is possible to create such automations in a generic way so that they will work with ANY entity of the same type. You won’t have to copy your automations for each and every entity or device. If you do it right, you create the flow in Node RED just once and it will kick in for each device of the same type or for all members of a defined list of devices / entities.

I think I achieve the same thing - to have an predefined schedule that can be manually overiden, but in a different way.

I use Schedy to control the climate in each room in the house, but the scheduling is influenced by a number of HA entities: one for the “season”, presence indicators for the occupants, motion sensors and a couple of manual overides. It would be simple to extend this to have a manual overide in each room - otherwise any manual overide lasts only until the next schedy-defined time.

@klacol From everything I have read here your idea seems to be the most practical. We need a way to define ml-sensors based on input-sensors. With this building block everything else could be created. Has anyone been doing more work in this direction?

1 Like

I totally understand the use case. I have this with light as well. Usually our lights are totally automated by motion sensors and illuminance sensors out- and inside. But sometimes we want to switch the light on or off regardless if an automation triggers or not. What I did is, to say alexa: “let the light off” and an automation deactivates the binary_sensor (binary_sensor.light_automation_office) for the specific room. It works but that is still uncomfortable due to the issue, that I have to activate the sensor after a certain time. I solved it in that way, that the command “let the light off” will be reactivated after 45 Minutes and the command “let the light permanently off” deactivates the sensor until the night. Every night at 23.45 all automation sensors are resetted. That is far away from being a smart and user intuitive solution :-).

By the way: You could give a min/max value to the input_number entity like this:

cooling_temperature:
  name: Zieltemperatur Klimageraet
  min: 16
  max: 23
  mode: box

That would save some code lines :wink:

Check out my project https://thesillyhome.ai/.
Our aim is to bring true automation to smart homes via ML.

The addon is available on Home Assistant and we are releasing continuosly!

1 Like