Inserting historical data into statistics as a continuous process

So yeah, that thing again… but with a slight twist :slight_smile:

Context: I’m building a device that uses Iridium Short Burst Data (SBD) to transfer sensor data to HA. The SBD service is built around small binary packets. Sending a packet costs money. Each packet has a given fixed size. You pay the same amount of money regardless of how much data you pack into the packet. So leaving parts unused is a (expensive) waste.

I have a sensor array that generates data in 5 minute intervals. My firmware packs several timestamped data points into a binary packet and uplinks it to the Iridium satellite network. On the terrestrial downlink side, a custom webapp running on a VPS receives the packet, unpacks it and I’m left with a list of around 3 to 5 timestamped sensor datasets that typically go back around 15 to 30 minutes. That time may vary per packet.

So. How to insert this into the HA database. I know this isn’t possible out of the box. I know there are a number of FRs about this. I know people have managed to import historical data directly into the DB, but mostly as a one-of offline process.

I need a reality / feasibility check from you guys who know the internals of the HA API and the recorder component before I dive into the framework (and having to learn Python on top of that). How involved would implementing this be. Where on the complexity scale are we here, from a simple undocumented API call that can insert historical data (doubt it, but one can dream), a more complex modification with potentially significant side effects on other components, or a complete rewrite of the recorder component ?

Assuming it’s the latter, how realistic is a continuous external insertion of data into the DB while HA is running (ie. locking DB writes from HA, inserting around 5 datapoints back in time, unlocking) every 15 to 30 minutes ? How badly would this affect HA ? Is HA going to get into a fit because it keeps internal caches that would have to be cleared, etc.

Just looking for some opinions if there is a realistic way to approach this without huge modifications to HA.

Thanks !