Taking Home Assistant to the next level. Home Assistant.. Assistant? - AI-powered Machine Learning for HA

Development
41

@dadaloop82, this is really interesting.
I’ve started collecting data in a SQLite database with the idea to create an algorithm that could predict next actions.
I used pyscript (that is much easier for me) for handling the database.
In order to limit the number of records I created two automations that trigger when the status of my devices change, one for covers and one for lights.
I’m now changing the database, trying to make it more meaningful, with more columns(data): each record should be a picture of the status of all entities in all the house.
So, now the database is really stupid, but once decided how to set it up I could share with you in order to let you evaluate your system with different data.
I’m open to suggestions for the set-up of the database.

EDIT:
@chrisvacc , sorry I missed to tag you.

1 Like
42

@chrisvacc Hi, I see a lot of interest with the posts but very little response form your side. Is this still active, i.e. are you still working on this?

43

Good morning everyone, sorry for my long absence.

I’ve been thinking a lot about applying AI to Home Assistant and I’ve done many experiments in this regard, but unfortunately they never gave me satisfaction.

I tried to change my point of view and wanted to express my experiences in this thread I just wrote in this Forum. I would be very pleased to receive your comments and proposals, your thoughts and concerns, in order to arrive together to a possible solution, in a constructive way.

Thank you!

44

I am a data-scientist with a background in Computer vision. Currently want to play with time-series data for my next project. Also it would be interesting to do a ML project in IOT/Smart Home area.

There are several ideas for me to try, but I am not HA user and have only a week long data from my setup (about 40 sensors and devices)

If you can share your data set it would be a great start

45

I know what you mean. I just started thinking of building a system with machine learning.but making it truly learn something is terrible complicated. Letting HA suggest automation based on stuff you do every day is fun but not really smart. Because it will only know this if you do the exact same thing every day. In which it’s easier to just write an automation for it and be done with it.

In my opinion AI needs to have a data set and being able make the connections on its own. So if I come out of bed one hour later it still needs to know what to do. And when I don’t work at home on Wednesday this week but a Thursday it needs to recognize that and act on it based on its findings. So it’s needs to recognize patterns that can differ in 1000s of ways but still knows what to do. I guess if you really want to go this far it’s simply impossible to write with a few lines of code. Google is spending 1000nds of man-hours a week on this and they can’t even set my alarm clock to go off earlier when there is a traffic jam. It’s very complicated. You have 1000s of variables that need to be checked every second

1 Like
46

@sygys I so totally agree with you. I am following this topic out of curiosity. But I often shook my head reading the high hopes that were expressed here. I am convinced that a neural network will rarely be able to infere the deeper motivations of our actions from the patterns of the sensors and actuators in our houses. It would surely produce occasional hits, but they would be embedded in a large number of wrong actions. These fails would be frustrating to all users and would totally ruin the acceptance of the system. Sorry for disenchanting you guys, but IMHO this is not feasible in the magical way that you are envisioning it.

47

Hey Team! I’ve started this same project earlier this year. Is there a slack channel that can be joined to jump into and participate the discussion and work?

48

@Jpsy At the moment im trying to find a way to being able to correct the system and then making the automations that did the wrong actions idle. So far it not very efficient and requires allot of additional lines of code to the automations. At the moment when an automation triggers it will turn on an input_boolean start the device and 10 seconds later turns the boolean off. The moment a device changes state another automation checks if the boolean went on. If so the system now knows the device was turned on by an automation. if a user turns the device back off and the boolean is off. the system knows the user turned a device on or of manually and another boolean will turn on that indicates that this device was turned on manually and this boolean stays on for the period i choose. This allready makes life easier if you have for example a AC that is programmed to function full automatic but you dont want it to turn on at the moment. The system will know the AC turned of by the user and wont turn back on anymore until the next day.

But this approach is terrible complicated to do for every device. I would love to see some sort of attribute for every device in home assistant which stores the last action and if it was triggered by an automation or manually. The AC is a hard component in this because we have an IR remote that turns on the AC without HA knowing. So thats why i did this approach at the moment and it works like a charm.

1 Like
49

Could you share this code?

50

Not sure this is possible directly in HA. But I think you can do this in Node RED using HA API calls. I didn’t try but it should be possible to add custom attributes to HA entities through the API.

You can use ESPHome and its IR component to let HA listen to the IR commands of your handheld in parallel. So HA becomes aware of all changes issued through the remote.

51

this is an automation that triggers the AC. it turns on the boolean that will indicate it did go on with an automation:

- id: '1628602498e629'
  alias: Airco woonkamer verwarmen en koelen
  description: ''
  trigger:
  - platform: time_pattern
    minutes: /5
  - platform: state
    entity_id: input_number.gewenste_temperatuur_woonkamer
    for: '00:00:05'
  condition: 
  - condition: not
    conditions:
      - condition: state
        entity_id: sensor.room_temp_thermostat_otgw
        state: 
          - 'unavailable'
          - 'unknown'
          - '0'
      - condition: state
        entity_id: sensor.tuin_weerstation_temperature
        state: 
          - 'unavailable'
          - 'unknown'
  action:
  - choose:
##### optie aan koelen#####
    - conditions:
      - condition: state
        entity_id: input_select.woonkamer_verwarming_modus
        state: 'koelen'
      - condition: time
        after: '06:00:00'
        before: '23:00:00'
      - condition: state
        entity_id: input_select.woning_status
        state: thuis
      - "{{states('input_number.gewenste_temperatuur_woonkamer')|float < states('sensor.room_temp_thermostat_otgw')|float }}"
      - condition: state
        entity_id: climate.airco_woonkamer_esp
        state: 'off'
      - condition: state
        entity_id: binary_sensor.woonkamer_dubbele_deur_sensor
        state: 'off'
      - condition: state
        entity_id: binary_sensor.achterdeur_sensor
        state: 'off'
      - condition: state
        entity_id: input_boolean.airco_woonkamer_handmatig
        state: 'off'
      sequence:
      - service: input_boolean.turn_on
        target:
          entity_id: input_boolean.airco_woonkamer_aan_uit_gezet_met_automation
      - service: climate.set_temperature
        data_template:
          hvac_mode: cool
          temperature: 22
        target:
          entity_id: climate.airco_woonkamer_esp
      - delay: 10
      - service: input_boolean.turn_off
        target:
          entity_id: input_boolean.airco_woonkamer_aan_uit_gezet_met_automation
      - delay: 5
      - service: climate.set_fan_mode
        target:
          entity_id: climate.airco_woonkamer_esp
        data:
          fan_mode: '1'
      - service: notify.discord_webhook
        data:
          message: 'Airco woonkamer op koelen'
      - delay: 5
      - service: select.select_option
        target:
          entity_id: select.airco_woonkamer_vertical_swing_mode
        data_template:
          option: up
##### optie aan verwarmen#####
    - conditions:
      - condition: state
        entity_id: climate.airco_woonkamer_esp
        state: 'off'
      - condition: state
        entity_id: binary_sensor.woonkamer_dubbele_deur_sensor
        state: 'off'
      - condition: state
        entity_id: binary_sensor.achterdeur_sensor
        state: 'off'
      - condition: state
        entity_id: input_boolean.airco_woonkamer_handmatig
        state: 'off'
      - condition: state
        entity_id: input_select.woonkamer_verwarming_modus
        state: 
          - 'hybrid'
          - 'elektrisch'
      - "{{states('input_number.gewenste_temperatuur_woonkamer')|float > states('sensor.room_temp_thermostat_otgw')|float }}"
      sequence:
      - service: input_boolean.turn_on
        target:
          entity_id: input_boolean.airco_woonkamer_aan_uit_gezet_met_automation
      - service: climate.set_temperature
        data_template:
          hvac_mode: heat
          temperature: "{{states('input_number.gewenste_temperatuur_woonkamer')|float + 5}}"
        target:
          entity_id: climate.airco_woonkamer_esp
      - delay: 10
      - service: input_boolean.turn_off
        target:
          entity_id: input_boolean.airco_woonkamer_aan_uit_gezet_met_automation
      - delay: 5
      - service: climate.set_fan_mode
        target:
          entity_id: climate.airco_woonkamer_esp
        data:
          fan_mode: '3'
      - delay: 5
      - choose:
        - conditions:
          - condition: state
            entity_id: binary_sensor.eettafel_person_motion
            state: 'off'
            for: '00:05:00'
          sequence:
          - service: select.select_option
            target:
              entity_id: select.airco_woonkamer_vertical_swing_mode
            data_template:
              option: down
        - conditions:
          - condition: state
            entity_id: binary_sensor.eettafel_person_motion
            state: 'on'
          sequence:
          - service: select.select_option
            target:
              entity_id: select.airco_woonkamer_vertical_swing_mode
            data_template:
              option: up
  mode: single

Then you have this to check when a state changed to the AC if it was an automation or not:

- id: '165347030dewc0016'
  alias: Check of airco woonkamer handmatig aangezet is
  description: ''
  trigger:
  - platform: state
    entity_id:
    - climate.airco_woonkamer_esp
    to:
      - 'cool'
      - 'dry'
      - 'heat'
      - 'off'
      - 'heat_cool'
    from:
      - 'cool'
      - 'dry'
      - 'heat'
      - 'off'
      - 'heat_cool'
  condition:
  - condition: state
    entity_id: input_boolean.airco_woonkamer_aan_uit_gezet_met_automation
    state: 'off'
  action:
  - service: input_boolean.turn_on
    data: {}
    target:
      entity_id: input_boolean.airco_woonkamer_handmatig
  mode: single

This will turn on a boolean to set it to manual when it was a manual operation.

And last i have an automation that controls the AC once its on but only when not manually turned on:

- id: '162aanpasse4564gerfn8629'
  alias: Airco woonkamer aanpassen tijdens koelen
  description: ''
  trigger:
  - platform: time_pattern
    minutes: /3
  - platform: state
    entity_id: input_number.gewenste_temperatuur_woonkamer
    for: '00:00:05'
  condition: 
  - condition: state
    entity_id: climate.airco_woonkamer_esp
    state: 'cool'
  - condition: state
    entity_id: input_boolean.airco_woonkamer_handmatig
    state: 'off'
  - condition: state
    entity_id: input_select.woonkamer_verwarming_modus
    state: 'koelen'

  action:
  - choose:
##########################################
##### optie 1 temp lager dan gewenst #####
##########################################
    - conditions:
      - condition: state
        entity_id: input_select.woning_status
        state: thuis
      - condition: template
        value_template: >
          {% set gewenste_temp = states('input_number.gewenste_temperatuur_woonkamer') | float %}
          {% set binnen_temp = states('sensor.room_temp_thermostat_otgw') | float %}
          {% set verschil = gewenste_temp - binnen_temp %}
          {% if (verschil > 0) %}
            true
          {% endif %}
      sequence:
##### TEMPERATUUR #####
      - service: climate.set_temperature
        data_template:
          hvac_mode: cool
          temperature: >
            {% set temperatuur = states('input_number.gewenste_temperatuur_woonkamer')|float +1 %}
              {% if temperatuur > 30 %} {{ 30 }}
            {% else %} 
              {{states('input_number.gewenste_temperatuur_woonkamer')|float +1 }}
            {% endif %}
        target:
          entity_id: climate.airco_woonkamer_esp
##### DELAY #####
      - delay: 5
##### PRESETMODE #####
      - choose:
        - conditions:
          - condition: state
            entity_id: climate.airco_woonkamer_esp
            attribute: preset_mode
            state: 
              - Normal
              - Powerful
          sequence:
          - service: climate.set_preset_mode
            target:
              entity_id: climate.airco_woonkamer_esp
            data:
              preset_mode: Quiet
        default: []

###########################################
##### optie 2 temp gelijk aan gewenst #####
###########################################
    - conditions:
      - condition: state
        entity_id: input_select.woning_status
        state: thuis
      - condition: template
        value_template: >
          {% set gewenste_temp = states('input_number.gewenste_temperatuur_woonkamer') | float %}
          {% set binnen_temp = states('sensor.room_temp_thermostat_otgw') | float %}
          {% set verschil = gewenste_temp - binnen_temp %}
          {% if (verschil == 0) %}
            true
          {% endif %}
      sequence:
##### TEMPERATUUR #####
      - service: climate.set_temperature
        data_template:
          hvac_mode: cool
          temperature: >
            {% set temperatuur = states('input_number.gewenste_temperatuur_woonkamer')|float %}
              {% if temperatuur < 16 %} 
                {{ 16 }}
              {% elif temperatuur > 30 %} 
                {{ 30 }}
              {% else %} 
                {{states('input_number.gewenste_temperatuur_woonkamer')|float }}
              {% endif %}
        target:
          entity_id: climate.airco_woonkamer_esp
##### DELAY #####
      - delay: 5
##### PRESETMODE #####
      - choose:
        - conditions:
          - condition: state
            entity_id: climate.airco_woonkamer_esp
            attribute: preset_mode
            state: 
              - Normal
              - Powerful
          sequence:
          - service: climate.set_preset_mode
            target:
              entity_id: climate.airco_woonkamer_esp
            data:
              preset_mode: Quiet
        default: []

##################################################
##### optie 3 temp 0 tot 1 graad hoger dan gewenst #####
##################################################
    - conditions:
      - condition: state
        entity_id: input_select.woning_status
        state: thuis
      - condition: template
        value_template: >
          {% set gewenste_temp = states('input_number.gewenste_temperatuur_woonkamer') | float %}
          {% set binnen_temp = states('sensor.room_temp_thermostat_otgw') | float %}
          {% set verschil = gewenste_temp - binnen_temp %}
          {% if (verschil < 0 ) %}
            true
          {% endif %}
      sequence:
##### TEMPERATUUR #####
      - service: climate.set_temperature
        data_template:
          hvac_mode: cool
          temperature: >
            {% set temperatuur = states('input_number.gewenste_temperatuur_woonkamer')|float -4 %}
              {% if temperatuur < 16 %} {{ 16 }}
            {% else %} 
              {{states('input_number.gewenste_temperatuur_woonkamer')|float -4 }}
            {% endif %}
        target:
          entity_id: climate.airco_woonkamer_esp
##### DELAY #####
      - delay: 5
##### FANMODE #####
      - choose:
        - conditions:
          - condition: not
            conditions:
            - condition: state
              entity_id: climate.airco_woonkamer_esp
              attribute: fan_mode
              state: '1'
          sequence:
          - service: climate.set_fan_mode
            target:
              entity_id: climate.airco_woonkamer_esp
            data:
              fan_mode: '1'
        default: []
##### DELAY #####
      - delay: 5
##### PRESETMODE #####
      - choose:
        - conditions:
          - condition: state
            entity_id: climate.airco_woonkamer_esp
            attribute: preset_mode
            state: Quiet
          sequence:
          - service: climate.set_preset_mode
            target:
              entity_id: climate.airco_woonkamer_esp
            data:
              preset_mode: Normal
        default: []
    default: []

This is only 3 of the 15 automations i currently have for controlling our 2 Ac’s

1 Like
52

I got all of this. What I wanted to tell you is, that it is possible to create such automations in a generic way so that they will work with ANY entity of the same type. You won’t have to copy your automations for each and every entity or device. If you do it right, you create the flow in Node RED just once and it will kick in for each device of the same type or for all members of a defined list of devices / entities.

53

I think I achieve the same thing - to have an predefined schedule that can be manually overiden, but in a different way.

I use Schedy to control the climate in each room in the house, but the scheduling is influenced by a number of HA entities: one for the “season”, presence indicators for the occupants, motion sensors and a couple of manual overides. It would be simple to extend this to have a manual overide in each room - otherwise any manual overide lasts only until the next schedy-defined time.

54

@klacol From everything I have read here your idea seems to be the most practical. We need a way to define ml-sensors based on input-sensors. With this building block everything else could be created. Has anyone been doing more work in this direction?

1 Like
55

I totally understand the use case. I have this with light as well. Usually our lights are totally automated by motion sensors and illuminance sensors out- and inside. But sometimes we want to switch the light on or off regardless if an automation triggers or not. What I did is, to say alexa: “let the light off” and an automation deactivates the binary_sensor (binary_sensor.light_automation_office) for the specific room. It works but that is still uncomfortable due to the issue, that I have to activate the sensor after a certain time. I solved it in that way, that the command “let the light off” will be reactivated after 45 Minutes and the command “let the light permanently off” deactivates the sensor until the night. Every night at 23.45 all automation sensors are resetted. That is far away from being a smart and user intuitive solution :-).

56

By the way: You could give a min/max value to the input_number entity like this:

cooling_temperature:
  name: Zieltemperatur Klimageraet
  min: 16
  max: 23
  mode: box

That would save some code lines :wink:

57

Check out my project https://thesillyhome.ai/.
Our aim is to bring true automation to smart homes via ML.

The addon is available on Home Assistant and we are releasing continuosly!

1 Like
58

Or send a prompt for edit or optimization :relaxed:

59

I am very much in for this topic. Most of my automation are around energy and water which are the most costly and scarce resources which I would like to optimise. Most of my automations are build on the basis of my own intelligence :laughing: and around energy and water.

My house is not of the grid but I have my own water well and I use solar power for heating and storing energy. In the future I will install photovoltaic panels as well and no doubt about an electric car.

Before you know you are in a spaghetti of scenes and data on which I like to make the most optimised intelligent automationS. I am not a developer at all but I can see the logic. All in all very interested as it will save money and resource if applied well!

1 Like
60

If anyone has seen the way chat-gpt can write python there is an interesting project where they are trying to opensource it for all manner of uses.

As a community you could prob feed chat-gpt home-assistant Q+A and on training let it run the show for you.
There is a current crop of some pretty amazing AI such as Whisper that will run on consumer hardware but your talking more Apple M1 than Pi4 but its not all doom and gloom as inference is highly spurious so many models can share a single hardware brain in a textbook client/server mode.

TFL-micro is sort of more node level sensor stuff but framwork doesn’t matter that much as the models often can be converted with Onnx getting a lot of love as a general purpose endpoint.
We are getting to the stage with a home HAL aka 2001 style.

1 Like