Advanced data analysis / Scatter plot data / Furnace and A/C monitor

Configuration
1

I’d like to monitor the performance of my furnace and A/C. Also I’d like to explore the capabilities of HA. And I very much love to analyze data.

Bringing this all together, I’d like to see a plot of how efficiently my house is heating (or cooling) and trigger alerts if it’s performing poorly.

For example, if the front door is open and it is taking longer to heat the house, or if the furnace igniter is not working and the call for heat is active but the house is not warming up, I’d like to be made aware of those scenarios.

Here’s the steps I’m thinking, to be executed after each cycle of heat or cooling is finished:

  1. Determine how long the furnace was heating (or cooling)
  2. Determine the indoor temperature delta during the cycle
  3. Calculate the temp change rate (e.g. °F/min)
  4. Determine the average outside temperature during the cycle
  5. Record a data point for x=outside avg temp, y=temp change rate
  6. Update a trend line (linear or polynomial regression) based on those data points

Once the data set is mature enough, I’d want to implement the alert functionality:

After a predefined duration during a call for heat (or cooling), I’d like to
A. Compare the in process heating/cooling rate data point to the trend line calculated in #6
B. If error is greater than a predetermined amount, trigger a notification

Technically all the data I need for this analysis already exists in the recorder and could be exported to tools I’m familiar with (such as Excel or Matlab) for parsing and analysis, but obviously the benefit of HA is automation.

This will obviously be a lot of work to implement. I’m looking for some high level suggestions on how to attack it. As I think through options, I’m coming up with solutions for only part of the problem and it is essentially a large number of template sensors (they’d all contain float values, updated upon the completion of each call for heat or cool)

So even once those sensors are implemented, I’m lost as to how I could analyze the data and feed it back into an automation.

I’m open to all methods and tools (node red, grafana, etc) so looking for some pointers on how to attack this. I hope the real benefit of creating this is for me to learn more about HA, as opposed to just implementing a notification. So comments such as “go read about X” are most definitely appreciated.

1 Like
2

You can possibly do some of what you want within HA using some of it’s tools like SQL sensor, sensor maths, integration, filters and such to create sensors and then scripts and automations.

That said, I try to do anything even slightly step wise complicated or computational outside of the HA process. I don’t think doing a lots of maths within HA is the best use case.

I’m a python guy, so you will hear my ideas pretty much go that route. I think many have found success with using Node Red, and I think you can create sensors within Grafana as well. However, for me, I have poked around a bit with these, but they are two more systems/languages that I have to maintain proficiency in for my ‘hobby’. As I said, I am somewhat python proficient, so I’m happy there.

There are a couple python based systems that you can write fairly complex python code in, include almost any of the ‘batteries included’ python libraries and create Home Assistant sensors right in your code. All in a separate process space from Home Assistant, so no burden on your HA system and possible response lags that might create ‘sensor input’ from your significant other :wink:.

Appdaemon is one I have used with success. I run it in a docker container on same machine as my HA docker container.

To ‘abstract’ this even more, I tend to use MQTT as my communication bridge into HA, from my other ‘projects’. I often will create python code to do my processing, have it publish MQTT topics. These I consume as MQTT sensors in HA. This model has more limits/complexity to getting to direct HA data in the form of HA sensors. So, for your application, the Appdaemon route might have some benefits. When I get one of these python apps running to a ‘production’ level, I will containerize it and run it in Docker.

That said, my next development ‘opinion’ is getting at historical sensor data, this is where I use a SQL database. HA, by default, stores it’s historical sensor and event data in a SQLite database. It’s great that the data is there and is stored in the nice tabular form of a SQL database. However, SQLite is not really designed to robustly allow multiple access. I’m sure other more qualified developers will debate this, but I’m an amateur and again goal #1 is don’t screw up HA.

So, I use a Postgresql database for my Home Assistant recorder. It, like most of my other ‘tools’, runs in a docker container side by side with my HA container. So, now I have a robust database of my HA sensor and event history data that I can access in real time without any worry of causing issues with HA using the same database server for it’s Recorder function.

My next point on my ‘python route’ is to your point about the ‘analysis’ part. I download from Docker Hub the latest Python Jupyter data science container and use Jupyter to do my analysis steps that lead to creating a python app running in docker that will consume HA sensor data and create HA sensors. Again, since Jupyter is python, I am coding the same basic code I will use to access my HA data and create my HA sensors, but in an interactive environment with wonderful visualization tools for the data. Again, most of my projects access to HA data comes via SQL queries to the Postgresql database of HA’s Recorder tables. I do do a few direct accesses to HA sensors from my external code, but this is a tad more difficult, but do able even in Jupyter.

Good hunting, one of the powers of HA is its historic data and turning those into actionable home automation ‘recommendations’! Hope your project is a success!

3

Very interesting idea. I would think more data is also needed. Number of humans home? Other heat sources? (Stove/dryer running when AC is calling etc…).

Watching…

4

Thanks a lot for all the ideas. A python app via Appdaemon looks like a really promising option. I recently installed InfluxDB so I need to see if I can have a python app query that, which will leave my main HA database (I use MariaDB) untouched. I started learning python a few weeks ago for another reason, so this would help out my learning.

And just starting to browse through the docs, it looks like I can store the scatter plot data in a “User Defined Namespace” where it can survive restarts and therefore I don’t have to recreate the historical dataset every time the script runs. I just have to look at the most recent heating/cooling cycle and add that data point to the file. However I have to figure out how I can graph the data in that namespace. Perhaps I’ll just have to write it to a file which can then be read by grafana.

5

It will be interesting to see how much scatter there is in the data. Realistically I could probably get by with something as simple as looking for no increase in the temp after 1hr of heating, and skip all this work altogether. But I’m interested to actually see the data, and to teach myself how to do this.

1 Like