I’ve recently discovered Home Assistant and ESPHome. I LOVE how easy it is to setup, deploy, maintain an ESP based device! I have an existing esp32 system I created and I’d like to see if moving it to HA + ESPHome is possible or a good idea.
I’ve got a unique situation where I’ve got some data logging and maybe switch control in the future. This is at a remote building with flakey internet connectivity and multi-day power outages. With limited physical access and week long power outages in the winter, reliability and power management are big concerns. Getting timely access to the old data once power is restored is important. eg So I can know if I’ll have to deal with frozen/cracked pipes. The alternative is to drive for 3 hours (one-way), inspect the cottage, take another two+ hours to town & back for supplies, fix things, then another 3 hours home or stay overnight in a freezing cold building. The bottom line is that having remote monitoring with access to ‘old’ data collected while offline is a huge time saver for me.
I coded an esp32 system that collects data, uses a SD card to save data (long-term backup record) & caches when needed, then uploads to the internet when it’s available. I’ve got about 35 entities, with an average of 10min between collection intervals (starts at 5min when on-battery, increase interval as outage gets longer). That’s 3360 data points/day, 23520/week. With 4-byte floats, I need at least 100kb for cached data, ideally 2-4x that so I don’t overflow storage if there’s a longer outage. That’s far too large for RTC slow ram.
It looks like architecture for ESPHome + HA is to forward data to HA immediately. Is there way to cache data on the ESP device? Will this work for a week or two?
Options I can see:
Cache data on ESP. Is this possible now? Maybe write my own module? (new process vs customize mqtt w/cache; look at sd card from Serrette Nicolas or Mnark)
Adding a ‘remote’ HA server (or mqtt bridge/cache??). Power is a non-trivial problem… I’d have to figure out a way to keep a RasPi or something powered for a week or two in the winter (~-30 C).
Use my existing code, but change to MQTT (or api??) and use HA for a better UI & backend. Probably easiest in the short term, but there’s no benefit to others. Updates are a pain.
Is there a better way of doing this? Is there something I’ve missed? Is there another approach to take?
How do you collect data when power is off? Do you have a UPS for that?
I would look for a solution using MQTT on one end and MQTT plus HA on the other. Let MQTT take care of storing the data.
MQTT has the (configurable) ability to store and forward data and catch up later when a connection is up. Whether your buffer size is large enough is the issue.
Low temperature tolerant batteries in a UPS may be another issue.
What are you measuring/controlling? Can you localise your smarts to the remote site altogether and grab the data later, or not at all?
But, this is well outside of what is normal for esphome and Home Assistant. There have been some threads around with people asking similar questions. This is not something that is easy to do. It will involve custom code.
It seems HA doesn’t accept old data. I use InfluxDB directly (through MQTT and NodeRED actually) and it does support adding old data. So it is possible to do.
Based on the other thread some people consider it silly to want to do that.
Thanks for the feedback so far. To reply to some questions:
I’ve got a lipo battery powering the esp32. This lives in a semi-climate controlled area along with my water system. The ‘critical’ data is temperatures so I know if my water system is frozen after a power outage. Battery life hasn’t been an issue yet with the esp32: low power draw on sleep (~0.25mA), increasing sleep duration as battery life goes down. I’ve managed a 7-day outage before.
I have a working local cache system on my esp32 via SD card. When power returns, it goes through the cache and uploads old data w/timestamps.
My current ‘backend’ is a horrific proof of concept (post data to google form, populates spreadsheet, graphs on sheet). It ‘seemed’ like a good idea at the time, but should never have been used and has gone on for FAR too long. I now have live-data MQTT working on my dev/test system.
Fortunately, no polar bears invoved.
Thanks for the thread and tip about HA not accepting old data. That’s disappointing.
I guess I’m looking at a separate DB for the data…
Do you know if MQTT catch up works on ESPHome? I don’t see any configurable options for cache in the documentation.
I’m told that HA doesn’t accept old data. Do you have any experience otherwise?
Smarts don’t really matter during a power outage. Nothing to control. I’ve already got data cache & update later working with my custom code but was hoping to move to a more standardized system.
MQTT QOS and persistent settings, retained, and clean session setting combinations not working as desired?
Alternately, embed the time/date stamp into the packet along with the temperature and unravel it at the receiving end to insert it into your database.
MQTT may not be the ideal option for breaks of so long. 'Relay’ing SQLITE message queue buffering and subsequent updates on reconnection is probably outside the scope of ESPHome.
Consider alternate database solutions that may offer store and forward options.
Yes, using spreadsheets as databases is a very common computing sin - even the UK NHS during Covid was caught out when the row/column limits in Excel were reached. Most corporate accounting divisions are guilty of this, nary a valid Microsoft Access licence across the whole division. Even sharebrokers who move billions in seconds using flaky Excel macros.
In your situation, I’m trying to imagine the traffic overheads when connection to Google Sheets is actually established. Yes, write your data out on csv/text format, but don’t use a spreadsheet as an intermediate to import it into your database.
First, I appreciate your thoughts on this. Happy New Year!
I’m new to MQTT. I haven’t tested QOS w/ persistent sessions yet. Even if QOS 1 or 2 and persistent sessions work the way I think they should work, it looks like they will not help for my case due to limited resources on the esp32. The esp32 MQTT client would need save persistance to SD or similar. ESPhome doesn’t have robust SD support or MQTT options for the persistent storage location. I’ve looked at a couple Arduino packages haven’t found one that appears to support storage backed persistence.
I think MQTT persistent sessions is a dead-end for me.
HA won’t work well with it - timestamps data when it arrives, not when collected.
ESPHome doesn’t appear to support it
Assorted Arduino MQTT libs don’t appear to support it
I don’t really want to do a deep dive into MQTT.
Do you know of any MQTT implementations on esp32 that can manage a 100kb-500kb backlog on persistent storage?
Do you know of any alternate DB solutions that will work on esp32 with a 100kb-500kb backlog on persistent storage?
MQTT for data transfer is YEARS ahead of my original PoC code. Right now, it looks like the way to go for data transfer. Managing the backlog on my own is fine - I already have code for that. Adding the collection timestamp for each entity in MQTT on my system is very doable. Unravelling it is what I’ll need to figure out.
It’s looking like MQTT to Telegraf with it’s merge plugin (current data into HA(still needed?) & DB, old data merged into DB) could be an option with VictoriaMetrics (or similar). Then use Grafana for the front end?
Sigh. I was really hoping to use an existing system rather than try to figure out how to use & integrate different systems at ~6 layers (mqtt client, mqtt broker, ETL, DB, UI1 visualization, UI2 automation?).
WTF - 6 layers each with multiple options?? As someone who doesn’t deal with this for a living, the number of permutations & combinations is overwhelming. My personal interest project is starting to sound a lot more like work now.
