Introducing ... TheSillyHome , a Homeassistant Machine Learning (AI) Addon

Nova · May 13, 2022, 6:39am

Motivation

Smart homes aren't very smart and definitely not user friend. I'm sure all of us can attest to the convoluted routines we have had to create to get working 50% of the time. In some ways, these static routines also adjust my behaviour.

How it works (as it stands)

Install with:

Container Rep

Addon repo

Data extraction

Homeassistant stores state data. This step extracts and parses this data into a ML trainable format (hot encoded of categorical values, constant status publish vs state changes etc..).

The end output frame is munged to show for each state published for an actuator, the state of other sensors and actuators.

The data is structured in a csv this: actuators, states, created, duplicate, senor1_state, sensor1_state, sensor2_state, sensor2_state, sensor3_state ...

Learning model

For each actuator, the sklearn Decision Tree model is applied over the dataset with the state as the output vector and the rest as the feature vector.

Appdaemon Execution

For ease of deployment, the decision was made to leverage Appdaemon in order to use the prediction models created in real time! For each sensor change there is a request to predict the new states for actuators and executes them.

Supported features

Model training
Prediction execution
Model Performance Dashboard
Retraining UI

Roadmap features currently in development

Teaching by doing - If the addon performs a bad action, you should be able to teach it on the spot to not do it again.
Centralized DB - Central cloud DB on AWS where all HA users are welcome to dump state data.
Base AI - Base AI Layer to help make models more accurate. Requires Centralized DB

I’m committed to devote significant time on this so don’t be shy to provide any suggestions. Please install and let me know if you face issues!

nickrout · May 13, 2022, 7:54am

Good on you for wanting to implement this. ~~Hard to contribute or even comment with no code though.~~

OK I see there is a link to your github in your MY link. I think it’d be good to include a simple github link so people can look at your repo BEFORE they are invited to include your repo on their machines.

lotohov · May 13, 2022, 1:58pm

tested and got some errors

2022-05-13 14:46:53.067061 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:46:53.076542 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:46:53.077932 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:46.973747 INFO model_executor: binary_sensor.motion_sensor_158d0001e163f6 is off
2022-05-13 14:48:47.063828 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:47.067496 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:48:47.071719 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d0001e163f6', 'new_state': 'off', 'old_state': 'on', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:48:47.073454 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:47.075359 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:48:47.076601 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:49.520628 INFO model_executor: binary_sensor.motion_sensor_158d000405931b is off
2022-05-13 14:48:49.582679 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:49.583728 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:48:49.585283 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d000405931b', 'new_state': 'off', 'old_state': 'on', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:48:49.586661 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:49.590855 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:48:49.591806 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:52.646275 INFO model_executor: binary_sensor.motion_sensor_158d0000f461c1 is off
2022-05-13 14:48:52.708525 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:52.709363 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:48:52.711081 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d0000f461c1', 'new_state': 'off', 'old_state': 'on', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:48:52.712393 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:48:52.713839 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:48:52.714748 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:17.191740 INFO model_executor: binary_sensor.motion_sensor_158d000405931b is on
2022-05-13 14:52:17.396325 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:17.399561 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:52:17.401960 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d000405931b', 'new_state': 'on', 'old_state': 'off', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:52:17.404599 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:17.407698 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:52:17.410469 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:20.827055 INFO model_executor: binary_sensor.motion_sensor_158d0000f461c1 is on
2022-05-13 14:52:20.988936 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:20.993888 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:52:21.010904 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d0000f461c1', 'new_state': 'on', 'old_state': 'off', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:52:21.023520 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:52:21.026209 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:52:21.035738 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:16.893576 INFO model_executor: binary_sensor.motion_sensor_158d000405931b is off
2022-05-13 14:54:16.967735 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:16.969133 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:54:16.970424 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d000405931b', 'new_state': 'off', 'old_state': 'on', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:54:16.971616 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:16.973187 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:54:16.974509 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:20.619086 INFO model_executor: binary_sensor.motion_sensor_158d0000f461c1 is off
2022-05-13 14:54:20.742148 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:20.743833 WARNING model_executor: Unexpected error in worker for App model_executor:
2022-05-13 14:54:20.745577 WARNING model_executor: Worker Ags: {'id': 'aec6536480c94565ad1addb89664e812', 'name': 'model_executor', 'objectid': 'f3ee871b007144678c5e02fbd08e339f', 'type': 'state', 'function': <bound method ModelExecutor.state_handler of <model_executor.ModelExecutor object at 0x7f789e0610>>, 'attribute': 'state', 'entity': 'binary_sensor.motion_sensor_158d0000f461c1', 'new_state': 'off', 'old_state': 'on', 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2022-05-13 14:54:20.748206 WARNING model_executor: ------------------------------------------------------------
2022-05-13 14:54:20.750578 WARNING model_executor: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/appdaemon/threading.py", line 917, in worker
    funcref(
  File "/appdaemon/apps/model_executor.py", line 49, in state_handler
    if (sensor + "_" + true_state) in df_sen_states.columns:
TypeError: can only concatenate str (not "NoneType") to str
2022-05-13 14:54:20.752069 WARNING model_executor: ------------------------------------------------------------

Nova · May 13, 2022, 7:02pm

Thanks for the feedback @nickrout . Have added the Git repo link!

Cao_Hoa · May 13, 2022, 11:53pm

do you have a docker-compose version for homeassistant I’m quite happy with your project

Nova · May 14, 2022, 12:21am

@lotohov Thanks for testing this out!
Unfortunately was blocked by this issue Hello_world example addon from developer docs stopped working (s6 overlay issue?) so could not test out the fix for your issue yet.

Could you share your config on the add-on and your states db if possible?

I am actually out on holiday next week but will try to get it fixed once I’m back.

lotohov · May 14, 2022, 12:09pm

give me tables u need iw ill try to export from mysql

actuactors_id:
  - light.bathroom
  - light.entrance
  - light.kabinet
  - light.kitchen
  - light.kitchen_strip
  - light.spalnaya
  - light.toliet
  - light.zal
sensors_id:
  - binary_sensor.motion_sensor_158d0001e163f6
  - binary_sensor.motion_sensor_158d0001d6683f
  - binary_sensor.motion_sensor_158d0000f461c1
  - binary_sensor.motion_sensor_158d000405931b
  - binary_sensor.sm_n975f_zariadka_ustroistva
  - binary_sensor.vibration_158d0002a4f516
  - person.lotohov
  - person.irishka
  - person.nastya

lotohov · May 14, 2022, 12:31pm

In this setup, we have a pip installable repo that hosts all the code GitHub - lcmchris/thesillyhome-addon-repo: add on repo for thesillyhome. This is installed onto the addon container and is built onto the addon-image. This addon uses the Appdaemon to control devices. This decision was made due to the simplicity of its implementation

Cao_Hoa · May 14, 2022, 3:53pm

I don’t use the Home Assistant Supervised version
but I use this version of Home Assistant Core, so I didn’t add it like you

Nova · May 17, 2022, 7:03am

Hi @lotohov
Apologies for the late reply - am currently out enjoying the sun in Gran Canaria!
The table I need is the “states” table.

You can use sqlworkbench or equivalent to do it!

Thanks,
Chris

Nova · May 17, 2022, 7:24am

Hi @Cao_Hoa
Thanks for expressing interesting in the project and I’ll try my best to support you.
The main reason for the lack of support for other installations types is that I personally don’t have those setup so it will be quite difficult for me to test it out.

Let me know if you any success converting this to a container but feel free to add a enhancement request onto the Github Repo for this.

juan11perez · May 18, 2022, 2:58pm

@Nova
interesting. I also run HA in docker so addon wont work.

i notice you load the appdaemon app into your ai image. Are you building an instance of appdaemon on your image itself?

I run appdaemon on a separate container, So I can add the app in the normal appdaemon insstance. Is this app designed to communicate with your image/python script?

I’m an enthusiast with some docker knowledge, so looking at your repo your dockerfile is already there. Would creating the app in appdaemon separately and building your image without it work?

Thank you

Nova · May 20, 2022, 7:26am

Hi @juan11perez
Thanks for taking interest!

The addon runs pip install appdaemon and also cp appdaemon /appdaemon to copy over the configuration.
You can indeed add the app but you will also need to pip install thesillyhome for the data extraction and sklearn training.
It probably doesn’t work straight out of the box. The reasoning to choose this path (instead of a pure appdaemon app) was that I thought addon installation is just much easier. It would be good if you can give me examples of addon repos that also has support for docker installations. I can try to build support for this.

juan11perez · May 20, 2022, 9:08am

@Nova thank you for your response.
Not sure if I’m missing it, but I dont see pip install appdeamon in the dockerfile.
I also noticed in some files it needs maria db. I think instructions need a bit more detail.

There’s popular addon for face recognition/identification called doubletake.
It started as a docker only and he subsequently developed the addon.

Nova · May 20, 2022, 3:33pm

Thanks for the example will take a look at it.

It should only require a sql-like db. I can definitely always add more details on the setup!!

The appdaemon install is from the requirements.txt under thesillyhome/requirements.txt at 028dad3ad3675c2f3fd6572a666fee7d005b6335 · lcmchris/thesillyhome · GitHub . Definitely not a good place to put it . Will see the best way to make this work for everyone.

Aim to publish something by next week! Keep an eye out.

juan11perez · May 20, 2022, 4:41pm

thank you very much for considering it.

Nova · May 31, 2022, 7:22am

Hi @juan11perez @Cao_Hoa
Just to update that I’m in the middle of changing the structure of the repo and making it into a standalone docker container. Much thanks for sharing double-take! To do this I also had to add a frontend so that the config can be added easily (which is why its taking a bit of time).

I do see the benefit of having this as standalone docker now ::

as Pi will probably not be able to handle training models once the dataset gets large or once fancier models are used. Great suggestion!
Easier development as I can test the results much quicker with a lighter weight image.

I am also going to make thesillyhome python package install locally in Docker (rather than pypi), just to make my life easier for now on the deployment side.

Should be ready by Friday for you guys to test!

juan11perez · May 31, 2022, 9:24am

Thank you for considering the feedback and your efforts.

Cao_Hoa · May 31, 2022, 1:40pm

Thank you for building a standalone docker project

Nova · June 5, 2022, 9:03pm

Hi @juan11perez and @Cao_Hoa ,

I’ve created the container repo here.

Unfortunately nothing really works yet with the new setup but it would be awesome if you guys could at least test out the docker-compose .
It should at least:

Start a node server listening on port 2300. You can access it on localhost:2300 but nothing on it works. I am a frontend noob to trying to get grips with how I can use it to trigger sh, py scripts, edit config and such.
Kicks off the processing but the default config is set to my instance (so should error out for you). You can try to edit the hesillyhome_src\data\config\options.json file with your instance. What is the best way to keep the config sustained through different docker runs?

Any advice would be sweet but will keep powering on!