Data Retention

It is part of Default Config. You have that instead.

There is no way to do this. There is one retention setting for everything.

Hmm… okay.
So, to edit Recorder for everything do I need to remove default config or can I just add

recorder:
    purge_keep_days: 366

to my config and it will overwrite?

I’ve picked 366 as I’d like to keep a year ideally for everything - you say don’t set it longer than 30 tho… why?

Is there no way to use an exclude on the purge?

Something like (psudo):

recorder:
    purge_keep_days: 366
       exclude:
          entities:
                - inputboolean.mysensor

No there isn’t. Just follow the recorder documentation. There are no hidden features.

Yes. Though I would not recommend retaining that much. Your database will likely start to become very sluggish.

On my setup I have a SSD with a Raspi4. I want to keep 30 days of history. So if I understood correctly, I just add in the configuration.yaml the following.

recorder:
  purge_keep_days: 30

Or do I have to add this under the default_config: like this:

default_config:
  
  recorder:
    purge_keep_days: 30

The first one.

1 Like

Not really. It does in fact work with the builtin db, it’s just really badly documented. Basically HA uses two different history engines, both built in: the ‘normal’ history and the long term statistics. Combining both, you’re able to achieve what you need without using additional external DBs.

The history is more precise and will use more memory and bandwidth. The DB engine itself (sqlite) is actually pretty good and can handle larger databases rather well. The problem is that the recorder implementation in HA is very inefficient. So keep the purge_keep_days at a reasonable number. 30 days seems reasonable, if your DB is stored on a reliable high bandwidth storage medium (not an SD card !). Some people have reported using it up to 90 days without issues. YMMV.

Next, the long term statistics. They go back indefinitely, but they only work for certain type of entities: entities with measurement or total_increasing state class. They’re less precise than history, but enough to do long term comparison on mean, min/max, etc. They’re more optimized too. Unfortunately they’re really badly documented. Here’s some info about them, but it’s pretty technical.

9 Likes

Thank you.
One question more: For the statistics card I see data for more than 30 days, even if I did not change the purge_keep_days.
For example I can see the daily mean for a battery of a sensor for 170 days. But the sensor-data for the sensor is just visible for 10 days. (see pictures below).

How long are the values for the statistic saved, and where do I change it?

They’re kept forever. You cannot change it. It’s independent of purge_keep_days. It’s an entirely separate history.

2 Likes

Ah, thats good to know! Thank you for that info.
For my temperature statistics the daily min, max and mean is enough for me. I don’t need more.
The hourly seems to be saved also forever. But the 5-min-values not. Correct?

Yes, correct.

1 Like

Is there a way to create a statistics type entity via helpers?
I.e. at the moment I have an input_number but honestly it could be anything that can be updated…

I’m not aware of any way to explicitely set the state class on input numbers. You may have to use a template sensor as an intermediate.

How would one go about doing that?
If it helps I already have a sensor which I’m using for UI representation (sensor.input_number_name) to remove the unneeded decimal point

I get this error when I try to add a data-source according to the video above

image

Hello, I would like to ask, if I want to retention only 2 sensor for 30 days, so this config correct?

recorder:
  purge_keep_days: 30
  include:
    entities:
      - sensor.sma_daily_yield_wh

And other sensor keep retention default 10 days?

That’s not possible. You can only set days for the whole database.

Petro, Thank You your answer.

When I’ve setup mariaDB on my nas, how do I change the retention to infinite?
Maybe I missed the reply here. But is it then 0 for infinit or do I need to fix that different?

and although this has gone stale, if i set up recorder to use an external DB does that include the history and the statistics… My DB runs on an external NVME server so im not worried about disk performance. I currently running an alternative HA solution nad its got 5 years worth of data, which i do occasionally go back to…

Just a question, reading this topic and thinking out of the box.
When I add a MQTT Sensor configured by Node-Red, for example:

The function node contains:
(this is a working setup I wrote to add any virtual sensor and used for more then a year. BTW dont just copy/paste because logTimeStamp is a custom variable.)

var logTimeStamp = global.get('logTimeStamp')||"No Timestamp";
var _topicBase = "homeassistant";
var _topicComponent = "sensor"
var _topicNodeId = "presence";
var _topicDeviceName = "detection";
var _topic = _topicBase + "/" + _topicComponent + "/" + _topicNodeId + "/" + _topicDeviceName
var _qos = 2;
var _retain = true;
var _payload = {
    "name": "Presence Detection",
    "icon": "hassio:eye-outline",
    "unique_id": "mqtt_presencedetection",
    "has_entity_name": "true",
    "node_id": _topicNodeId,
    "state_topic": _topic + "/state",
    "device": {
        "name": "Mqtt Security",
        "manufacturer": "MQTT Hass.io",
        "model": "Custom",
        "identifiers": ["Mqtt Security"],
        "sw_version": "20230803",
        "suggested_area": "Algemeen"
        }
    };
if (msg.payload === '' ){
     node.status({fill:"red",shape:"dot",text: logTimeStamp + " " + _topic});
     _topic = _topic + "/config";
     _qos = 0;
     _retain = false;
     _payload = '';
    } else {
     node.status({fill:"green",shape:"dot",text: logTimeStamp + " " + _topic});
     _topic = _topic + "/config";
    }    
var message = {
    topic: _topic,
    qos: _qos,
    retain: _retain,
    payload: _payload
};
return message;

The MQTT device is using (as an example) just four states: “home”, “away”, “sleep”, “holliday”.

The question, can I add, at the var _payload config, to get the four states to be stored in the long-term-statistics? :

¨device_class": "enum",
"options": {}

If so, how do I configure the options list and can this be a way to do it?
Any idea?