Purge is incorrectly removing historical data

Configuration
1

Background

I have some scripts Schedule to purge data using different days on configuration depending of the time I want to keep the data.
This purge scripts are scheduled weekly.
I expected this to remove old data on the DB

Problem
I realize my DB keep growing so I realize I was just doing a purge, but never a repack to actually “shrink” the dab.
I create a new script to “force” a REPACK:

  - action: recorder.purge
    metadata: {}
    data:
      repack: true

I executed and after some hour my DB decrease a lot, (from around 15000Mb to 5000Mb)
I do a quick check, and all looks ok. So I was so happy.

Then I realize I have lost all my old data: basically my older data is now for Aug. 16th (around 10 days ago), but I would like to keep some entity info for a year):
before repack
image

After repack
image

I restore the back up and data is back.

Just to be sure I did same procedure and check my data on every step and I am 100% sure data is removed by repack action, and not on any other purge action.

What I am missing there? I understand repack should ONLY shrink BD to take advantage of free space on the DB, but should never remove data, correct?

Any idea?

NOTE: I am also using influx DB ad Grafana for old data storage and consult, but I need to fully understand why purge is removing data when should not.

NOTE_2: I also create an issue on github, as per my understanding repack would never remove data, but let m ekno if make sense to only keep this.

thansk

2

Hi, anyone can help, at least to confirm if repack should remove data or not? Anything you see wrong on the script?

thanks

3

State data is kept foe 10 days (by default) then deleted automatically. Everything you have observed is working as intended.

4

Thanks! So I should add a bigger number of days to do the repack without loosing data, correct?

5

Do not set your keep days longer than 30 days or you may see reduced performance.

This will increase the size of your database by a factor of 3.

What are you actually trying to accomplish?

The database is repacked every month automatically. Why are you manually doing this?

6

Hi

i am tring to keep one year for some temp and power consume entities to be able to compare, not fro all entities…

I did not see any performance issue so far, and some entities are already restored for more than a year. As I told, for other entities, I used to purge data older than a week or a month… (i have my own purge scripts running every week.)

No, repack is not running automatically as I have auto-purge option disabled, this is why i what to manually run.

I will reconsider my purge retention policies one more time, but i do nto feel keeping old data just for a few entities should harm the system (i am using arround than 20% of my HD. less than 30% of RAM and 5% of CPu as avg)

7

Then give these sensors a state class and LTS will be kept forever. Do not set your recorder to retain more than 30 days.

Repack happens automatically first Sunday of the month.

1 Like
8

second

1 Like
9

Dammit I initially wrote:

Repack happens automatically first (second?) Sunday of the month.

But deleted that bit.

10

Heh, I don’t know why it’s the second sunday :man_shrugging: