I have a ESP device that pushes sensor data to HA that needs to be replaced, but I do not want to break the sensor data presented in the Energy tab.
As I see it the new device will be percived as an additional device rather than a replacement (new MAC, new device name etc).
Will it be sufficient to ensure that the entities have the same id, or do I need to go deeper, eg to device_id?
Before 2023.4 you could give a new entity the same entity_id as an old one and it would automatically pick up the old data. However 2023.4 introduced a database change where internally each entity now has a āmetadata_idā, and the state data is recorded against this id. You can see the impact of this on the updated documentation for the SQL Sensor, which now includes āmetadata_idā. A new entity will get a new āmetadata_idā, and therefore new history. I donāt know of any way to connect the old history to the new without tinkering in the database.
EDIT: Actually youāre referring to long-term data, but the principle still applies, as that also uses āmetadata_idā (different values than those for states I think though).
I made an attempt yesterday that did not go well - the new device got picked up as a cloned copy - but only partial, most of the entities actually got associated with the new device, as did the automation 
Lucklily I had a full backup from a few days back so that I could restore the config settings - no energy data was pulled during the brief test period so the underlying databases were not touched.
Considering the lifespan of some of the IoT at least I believe that there needs to be a method to seamlessly replace one device with another, the metadata_id is contraproductive, or at least not fully impact analysed, in this regards.
Did you ever find a solution for this?
Found your topic after I ran into the very same question @
Not to this date.
Unfortunately it slips out of mind as long as things are working as intended, and will probably come crashing back when/if next thing breaks.
On the other hand, my main purpose with HA is the everyday automation of my home, that I can have access to long term statistics might not be persieved as the as the core functionality of the system. In is not a feature I requested of the predecessor of my HA installation, so maybe I should just ditch that aspect from HAā¦
Well, people using at least the energy dashboard usually care a lot. As you said: as long as everything is working, fine - if you need to replace devices/sensors/ENTITIES, itās kinda important.
Currently, I believe it would take a fair amount of experimentation to discover a manual way to transfer an old deviceās history to a new one. Naturally, mistakes could easily corrupt the system so itās not something to do without a lot of time, backups, and patience (and even then, maybe not a production system).
The old device is registered with a unique_id (in the hidden entity registry) and an identifier (in the database).
The same thing is done for the new device.
You would have to set the new deviceās unique_id and identifier to the same values as the old device. Theoretically, the new device would then have the old deviceās long-term statistics.
Potential roadblocks to success:
I believe that if the old device is removed, its long-term statistics will also be removed. I have not confirmed this (I donāt want to experiment with my systemās data) but it would seem to make sense from a house-keeping standpoint. So the old device cannot be deleted before the new device is added.
The entity registry canāt have two devices with the same unique_id. I imagine the database (storing long-term statistics) wouldnāt like it either. Naturally, Home Assistant would need to be disabled while these changes were being done.
Wow, if not even you Taras, one of the most active users here (or even a mod, right?) knows thisā¦ I tend to say: no one knows this.
Iām really a bit (a lot of a bit tbh) annoyed about this. One change has been promoted as improvement/great new feature, while at the same time not providing a solution for the downside of this change. Now people have no more control over the history of entities, they are not interchangeable anymore - so in the end one actually LOOSES data (history / LTS), which is - depending on the point of view - exactly the opposite of the Home Assistant now keeps history when renaming entities :metal: announcement at 2023.4: Custom template macros, and many more new entity dialogs! - Home Assistant.
Thatās really bad because swapping entities saved me so much time and hassle in the past. Hopefully one will come up with a workaround solution (āthe manual wayā) or - no idea how - thereās a smart person adding a feature request to raise importance for and kick-start this (having a GUI feature in the dev tools / statistics section for example).
What would you recommend to make some progress here? Iām rather looking at the 10 seconds brainstorming idea of the GUI based āstatistics transfer featureā in the dev tools statistics section instead of risking my (one and only) productive system to see if thereās a way to actually hack HA to preserve LTS.
Really, guys, what do you do when devices die, e. g. when their sensors are part of the energy dashboard?
Not sure about devices, but on an entity level (as of until Core 2023.3): definitely no. They will sit and get rusty in the database forever. See WTH can't we fix recorder data easily using the UI (dev-tools section) - single data points + mass- changes? and Remove statistics of disabled (not deleted!) entities from recorder (took a deep dive into LTS fixes and deletions one year ago).
Run this and youāll probably be surprised how much rubbish is left in LTS while there is no corresponding data in states table anymore (deleted entities, left LTS):
SELECT
CASE WHEN entity_id IS NULL
THEN 'Orphaned'
ELSE 'OK'
END Issue,
entity_id,
statistic_id
FROM (
SELECT
b.entity_id,
a.statistic_id
FROM
statistics_meta a
LEFT JOIN 'states_meta' b
ON a.statistic_id = b.entity_id
GROUP BY
a.statistic_id -- we want to avoid duplicates
)
My small setup already has quite some of that (āorphaned statistics entities in recorderā):
By the way no idea what happened to https://community.home-assistant.io/t/statistics-fix-issue-large-amount-of-data/419728 - the URL is fine, but the thread is not available - did not survive the forums migration? Luckily it has been archived, at leastā¦ https://web.archive.org/web/20230401222744/https://community.home-assistant.io/t/statistics-fix-issue-large-amount-of-data/419728
Someone probably does know the right sequence of steps but not me, mostly because I have never had a need to try.
If you want a new entity to take over an old entityās long-term statistics, then thatās probably a good thing that their long-term stats remain untouched (otherwise, I agree itās not ideal to have āleftoversā in the database).
Regarding my use of the word ādeviceā, I probably should have said āentityā.
I apologize if Iām missing something, but does this procedure not keep the history and long term statistics data tied to the new sensor?
No, I donāt think this is still working (step 5).
Change the entity ID of the new entity to be identical to the old one (the one you just removed): (
sensor.my_new_energy_deviceāsensor.my_old_energy_device)
A few days ago I changed from ZHA to Zigbee2mqtt, and changed Zigbee coordinators at the same time, so I just went through this for each of my ~15 Zigbee devices. I never rebooted; but otherwise essentially followed these instructions. My long term statistics has carried over as far as I can tell.
Indeed this seems to be the way to go (following HA Energy Dashboard FAQ Ā· GitHub). Essentially it comes down to this (updated) post:
The entity migration approach has been changed after 2023.4.
The following way worked for me:
OS Version: Home Assistant OS 11.4
Home Assistant Core: 2024.1.5
The new process is:
1. Delete old entity (Settings > Device&services > Entities) It was in "Restored" status. Select the checkbox at the begining of the row!
2. Rename new entity to old entity_id -> click to the entity_id > settings > rename entity_id > update
3. New entity will inherit old entities history (no reboot needed, it is on the fly...)
IMPORTANT: Do not under any circumstances rename the old entity before you delete it, this will cause the history to follow the renamed name. (If you already renamed it, beacuse you followed the earlier process, you have to rename it back, than reboot, than follow the new process above. It worked me this way too.)
How will this work, if I donāt want to delete the old entity, but just rename it to <someid>_raw.
My use case is: I want put a filter āin frontā of one of my mqtt sensors to remove outliers, but without loosing LTS. So the new filtered sensor (a template sensor) will replace the <someid>_raw.