I have a ESP device that pushes sensor data to HA that needs to be replaced, but I do not want to break the sensor data presented in the Energy tab.
As I see it the new device will be percived as an additional device rather than a replacement (new MAC, new device name etc).
Will it be sufficient to ensure that the entities have the same id, or do I need to go deeper, eg to device_id?
Before 2023.4 you could give a new entity the same entity_id as an old one and it would automatically pick up the old data. However 2023.4 introduced a database change where internally each entity now has a “metadata_id”, and the state data is recorded against this id. You can see the impact of this on the updated documentation for the SQL Sensor, which now includes “metadata_id”. A new entity will get a new “metadata_id”, and therefore new history. I don’t know of any way to connect the old history to the new without tinkering in the database.
EDIT: Actually you’re referring to long-term data, but the principle still applies, as that also uses “metadata_id” (different values than those for states I think though).
I made an attempt yesterday that did not go well - the new device got picked up as a cloned copy - but only partial, most of the entities actually got associated with the new device, as did the automation 
Lucklily I had a full backup from a few days back so that I could restore the config settings - no energy data was pulled during the brief test period so the underlying databases were not touched.
Considering the lifespan of some of the IoT at least I believe that there needs to be a method to seamlessly replace one device with another, the metadata_id is contraproductive, or at least not fully impact analysed, in this regards.
Did you ever find a solution for this?
Found your topic after I ran into the very same question @
Not to this date.
Unfortunately it slips out of mind as long as things are working as intended, and will probably come crashing back when/if next thing breaks.
On the other hand, my main purpose with HA is the everyday automation of my home, that I can have access to long term statistics might not be persieved as the as the core functionality of the system. In is not a feature I requested of the predecessor of my HA installation, so maybe I should just ditch that aspect from HA…
Well, people using at least the energy dashboard usually care a lot. As you said: as long as everything is working, fine - if you need to replace devices/sensors/ENTITIES, it’s kinda important.
Currently, I believe it would take a fair amount of experimentation to discover a manual way to transfer an old device’s history to a new one. Naturally, mistakes could easily corrupt the system so it’s not something to do without a lot of time, backups, and patience (and even then, maybe not a production system).
The old device is registered with a unique_id (in the hidden entity registry) and an identifier (in the database).
The same thing is done for the new device.
You would have to set the new device’s unique_id and identifier to the same values as the old device. Theoretically, the new device would then have the old device’s long-term statistics.
Potential roadblocks to success:
I believe that if the old device is removed, its long-term statistics will also be removed. I have not confirmed this (I don’t want to experiment with my system’s data) but it would seem to make sense from a house-keeping standpoint. So the old device cannot be deleted before the new device is added.
The entity registry can’t have two devices with the same unique_id. I imagine the database (storing long-term statistics) wouldn’t like it either. Naturally, Home Assistant would need to be disabled while these changes were being done.
Wow, if not even you Taras, one of the most active users here (or even a mod, right?) knows this… I tend to say: no one knows this.
I’m really a bit (a lot of a bit tbh) annoyed about this. One change has been promoted as improvement/great new feature, while at the same time not providing a solution for the downside of this change. Now people have no more control over the history of entities, they are not interchangeable anymore - so in the end one actually LOOSES data (history / LTS), which is - depending on the point of view - exactly the opposite of the Home Assistant now keeps history when renaming entities :metal: announcement at 2023.4: Custom template macros, and many more new entity dialogs! - Home Assistant.
That’s really bad because swapping entities saved me so much time and hassle in the past. Hopefully one will come up with a workaround solution (“the manual way”) or - no idea how - there’s a smart person adding a feature request to raise importance for and kick-start this (having a GUI feature in the dev tools / statistics section for example).
What would you recommend to make some progress here? I’m rather looking at the 10 seconds brainstorming idea of the GUI based “statistics transfer feature” in the dev tools statistics section instead of risking my (one and only) productive system to see if there’s a way to actually hack HA to preserve LTS.
Really, guys, what do you do when devices die, e. g. when their sensors are part of the energy dashboard?
Not sure about devices, but on an entity level (as of until Core 2023.3): definitely no. They will sit and get rusty in the database forever. See WTH can't we fix recorder data easily using the UI (dev-tools section) - single data points + mass- changes? and Remove statistics of disabled (not deleted!) entities from recorder (took a deep dive into LTS fixes and deletions one year ago).
Run this and you’ll probably be surprised how much rubbish is left in LTS while there is no corresponding data in states table anymore (deleted entities, left LTS):
SELECT
CASE WHEN entity_id IS NULL
THEN 'Orphaned'
ELSE 'OK'
END Issue,
entity_id,
statistic_id
FROM (
SELECT
b.entity_id,
a.statistic_id
FROM
statistics_meta a
LEFT JOIN 'states_meta' b
ON a.statistic_id = b.entity_id
GROUP BY
a.statistic_id -- we want to avoid duplicates
)
My small setup already has quite some of that (“orphaned statistics entities in recorder”):
By the way no idea what happened to https://community.home-assistant.io/t/statistics-fix-issue-large-amount-of-data/419728 - the URL is fine, but the thread is not available - did not survive the forums migration? Luckily it has been archived, at least… https://web.archive.org/web/20230401222744/https://community.home-assistant.io/t/statistics-fix-issue-large-amount-of-data/419728
Someone probably does know the right sequence of steps but not me, mostly because I have never had a need to try.
If you want a new entity to take over an old entity’s long-term statistics, then that’s probably a good thing that their long-term stats remain untouched (otherwise, I agree it’s not ideal to have ‘leftovers’ in the database).
Regarding my use of the word ‘device’, I probably should have said ‘entity’.
I apologize if I’m missing something, but does this procedure not keep the history and long term statistics data tied to the new sensor?
No, I don’t think this is still working (step 5).
Change the entity ID of the new entity to be identical to the old one (the one you just removed): (
sensor.my_new_energy_device→sensor.my_old_energy_device)
A few days ago I changed from ZHA to Zigbee2mqtt, and changed Zigbee coordinators at the same time, so I just went through this for each of my ~15 Zigbee devices. I never rebooted; but otherwise essentially followed these instructions. My long term statistics has carried over as far as I can tell.
Indeed this seems to be the way to go (following HA Energy Dashboard FAQ · GitHub). Essentially it comes down to this (updated) post:
The entity migration approach has been changed after 2023.4.
The following way worked for me:
OS Version: Home Assistant OS 11.4
Home Assistant Core: 2024.1.5
The new process is:
1. Delete old entity (Settings > Device&services > Entities) It was in "Restored" status. Select the checkbox at the begining of the row!
2. Rename new entity to old entity_id -> click to the entity_id > settings > rename entity_id > update
3. New entity will inherit old entities history (no reboot needed, it is on the fly...)
IMPORTANT: Do not under any circumstances rename the old entity before you delete it, this will cause the history to follow the renamed name. (If you already renamed it, beacuse you followed the earlier process, you have to rename it back, than reboot, than follow the new process above. It worked me this way too.)
How will this work, if I don’t want to delete the old entity, but just rename it to <someid>_raw.
My use case is: I want put a filter “in front” of one of my mqtt sensors to remove outliers, but without loosing LTS. So the new filtered sensor (a template sensor) will replace the <someid>_raw.
And how about if I don’t want to completely stop using the “old” device?
Here is my use case: I have a smart plug that tracks my server’s energy usage. I want a new device (a Smart UPS) to take over the LTS from the smart plug, and continue tracking power data where the smart plug left off.
I managed to do this earlier today and it worked a charm - the data carried over with no issues.
BUT, I then want to re-commission the smart plug to track my DESK’S energy usage, starting from zero. The second I plugged the smart plug (Tasmota) back into an outlet, it messed up all the energy data in HA, re-linking it back to this smart plug - even though I had previously deleted it from HA, reset it to factory settings, and set it up from new via Tasmota web gui with a totally different name.
If anyone knows a workaround for this I’d appreciate it. There must be some device id somewhere at a device-level, which HA has picked up on and recognised as a pre-existing device? Does this mean you can’t ever re-use energy tracking devices for another use-case? If so this is quite a disappointing shortfall for the HA energy dashboard functionality.