Introducing ... TheSillyHome , a Homeassistant Machine Learning (AI) Addon

Motivation

Smart homes aren't very smart and definitely not user friend. I'm sure all of us can attest to the convoluted routines we have had to create to get working 50% of the time. In some ways, these static routines also adjust my behaviour.

How it works (as it stands)

Install with:
Open your Home Assistant instance and show the add add-on repository dialog with a specific repository URL pre-filled.

Addon repo

Python package repo

Data extraction

Homeassistant stores state data. This step extracts and parses this data into a ML trainable format (hot encoded of categorical values, constant status publish vs state changes etc..).

The end output frame is munged to show for each state published for an actuator, the state of other sensors and actuators.

The data is structured in a csv this: actuators, states, created, duplicate, senor1_state, sensor1_state, sensor2_state, sensor2_state, sensor3_state ...

Learning model

For each actuator, the sklearn Decision Tree model is applied over the dataset with the state as the output vector and the rest as the feature vector.

Appdaemon Execution

For ease of deployment, the decision was made to leverage Appdaemon in order to use the prediction models created in real time! For each sensor change there is a request to predict the new states for actuators and executes them.

Roadmap features currently in development

  • Controls - Add controls on UI to separate the 3 steps above.
  • Routine Extraction - extract simple rules from the AI model. Improves transparency of the AI.
  • Teaching by doing - If the addon performs a bad action, you should be able to teach it on the spot to not do it again.

Backlog

  • Model Performance Dashboard - helps monitor your Model accuracy
  • Centralized DB - Central cloud DB on AWS where all HA users are welcome to dump state data.
  • Base AI - Base AI Layer to help make models more accurate. Requires Centralized DB

I’m committed to devote significant time on this so don’t be shy to provide any suggestions. Please install and let me know if you face issues!

7 Likes

Good on you for wanting to implement this. Hard to contribute or even comment with no code though.

OK I see there is a link to your github in your MY link. I think it’d be good to include a simple github link so people can look at your repo BEFORE they are invited to include your repo on their machines.

tested and got some errors

2022-05-13 14:46:53.067061 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:46:53.076542 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:46:53.077932 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:46.973747 INFO model_executor: binary_sensor.motion_sensor_158d0001e163f6 is off
2022-05-13 14:48:47.063828 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:47.067496 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:48:47.071719 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d0001e163f6', 'new_state': 'off', 'old_state': 'on', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:48:47.073454 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:47.075359 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:48:47.076601 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:49.520628 INFO model_executor: binary_sensor.motion_sensor_158d000405931b is off
2022-05-13 14:48:49.582679 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:49.583728 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:48:49.585283 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d000405931b', 'new_state': 'off', 'old_state': 'on', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:48:49.586661 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:49.590855 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:48:49.591806 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:52.646275 INFO model_executor: binary_sensor.motion_sensor_158d0000f461c1 is off
2022-05-13 14:48:52.708525 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:52.709363 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:48:52.711081 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d0000f461c1', 'new_state': 'off', 'old_state': 'on', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:48:52.712393 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:52.713839 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:48:52.714748 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:17.191740 INFO model_executor: binary_sensor.motion_sensor_158d000405931b is on
2022-05-13 14:52:17.396325 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:17.399561 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:52:17.401960 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d000405931b', 'new_state': 'on', 'old_state': 'off', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:52:17.404599 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:17.407698 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:52:17.410469 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:20.827055 INFO model_executor: binary_sensor.motion_sensor_158d0000f461c1 is on
2022-05-13 14:52:20.988936 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:20.993888 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:52:21.010904 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d0000f461c1', 'new_state': 'on', 'old_state': 'off', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:52:21.023520 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:21.026209 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:52:21.035738 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:16.893576 INFO model_executor: binary_sensor.motion_sensor_158d000405931b is off
2022-05-13 14:54:16.967735 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:16.969133 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:54:16.970424 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d000405931b', 'new_state': 'off', 'old_state': 'on', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:54:16.971616 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:16.973187 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:54:16.974509 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:20.619086 INFO model_executor: binary_sensor.motion_sensor_158d0000f461c1 is off
2022-05-13 14:54:20.742148 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:20.743833 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:54:20.745577 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d0000f461c1', 'new_state': 'off', 'old_state': 'on', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:54:20.748206 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:20.750578 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:54:20.752069 WARNING model_executor: ------------------------------------------------------------

Thanks for the feedback @nickrout . Have added the Git repo link!

do you have a docker-compose version for homeassistant I’m quite happy with your project

@lotohov Thanks for testing this out!
Unfortunately was blocked by this issue Hello_world example addon from developer docs stopped working (s6 overlay issue?) so could not test out the fix for your issue yet.

Could you share your config on the add-on and your states db if possible?

I am actually out on holiday next week but will try to get it fixed once I’m back.

give me tables u need iw ill try to export from mysql

actuactors_id:
  - light.bathroom
  - light.entrance
  - light.kabinet
  - light.kitchen
  - light.kitchen_strip
  - light.spalnaya
  - light.toliet
  - light.zal
sensors_id:
  - binary_sensor.motion_sensor_158d0001e163f6
  - binary_sensor.motion_sensor_158d0001d6683f
  - binary_sensor.motion_sensor_158d0000f461c1
  - binary_sensor.motion_sensor_158d000405931b
  - binary_sensor.sm_n975f_zariadka_ustroistva
  - binary_sensor.vibration_158d0002a4f516
  - person.lotohov
  - person.irishka
  - person.nastya

In this setup, we have a pip installable repo that hosts all the code GitHub - lcmchris/thesillyhome-addon-repo: add on repo for thesillyhome. This is installed onto the addon container and is built onto the addon-image. This addon uses the Appdaemon to control devices. This decision was made due to the simplicity of its implementation

I don’t use the Home Assistant Supervised version
but I use this version of Home Assistant Core, so I didn’t add it like you

Hi @lotohov
Apologies for the late reply - am currently out enjoying the sun in Gran Canaria!
The table I need is the “states” table.

You can use sqlworkbench or equivalent to do it!

Thanks,
Chris

Hi @Cao_Hoa
Thanks for expressing interesting in the project and I’ll try my best to support you.
The main reason for the lack of support for other installations types is that I personally don’t have those setup so it will be quite difficult for me to test it out.

Let me know if you any success converting this to a container but feel free to add a enhancement request onto the Github Repo for this.

@Nova
interesting. I also run HA in docker so addon wont work.

i notice you load the appdaemon app into your ai image. Are you building an instance of appdaemon on your image itself?

I run appdaemon on a separate container, So I can add the app in the normal appdaemon insstance. Is this app designed to communicate with your image/python script?

I’m an enthusiast with some docker knowledge, so looking at your repo your dockerfile is already there. Would creating the app in appdaemon separately and building your image without it work?

Thank you

Hi @juan11perez
Thanks for taking interest!

  1. The addon runs pip install appdaemon and also cp appdaemon /appdaemon to copy over the configuration.
  2. You can indeed add the app but you will also need to pip install thesillyhome for the data extraction and sklearn training.
  3. It probably doesn’t work straight out of the box. The reasoning to choose this path (instead of a pure appdaemon app) was that I thought addon installation is just much easier. It would be good if you can give me examples of addon repos that also has support for docker installations. I can try to build support for this.

@Nova thank you for your response.
Not sure if I’m missing it, but I dont see pip install appdeamon in the dockerfile.
I also noticed in some files it needs maria db. I think instructions need a bit more detail.

There’s popular addon for face recognition/identification called doubletake.
It started as a docker only and he subsequently developed the addon.

1 Like

Thanks for the example will take a look at it.

It should only require a sql-like db. I can definitely always add more details on the setup!!

The appdaemon install is from the requirements.txt under thesillyhome/requirements.txt at 028dad3ad3675c2f3fd6572a666fee7d005b6335 · lcmchris/thesillyhome · GitHub . Definitely not a good place to put it :blush: . Will see the best way to make this work for everyone.

Aim to publish something by next week! Keep an eye out.

thank you very much for considering it.