InfluxDB - Change exclude to Include - Examples

Ok, so I wanted to put some effort in my ever growing influxdb which is now around 40Gb in size :-).

But like any other “tasks” in HA it’s not so simple.

I read about influxdb and I use the addon. I Assume I have v1 (not 2).
I have and exclude list now in recorder and in influxbd, mostly the same.
I tried to search how to cleanup influx, not found any helpful sources.
Then I tried to find out how to get “wanted” entities in influx:

  • only certain integrations (cannot find how to do this).

All these tasks… I end up in dead ends…

Thus I wanted to ask, just as a starter, can anyone who has an “include” list for influx please share it below?

Additionally - is it a good moment to start influxdb v2 ? Can 1 and 2 run in parallel?

This is my current config and added “include” config.
I am struggling to understand the rules of InfluxDB - Home Assistant

I expect that:

  1. the include will be included, then nothing else will be included.
  2. the exclude will all be extra excluded on top of the nothing else will be included.
influxdb:
# below the newly added entities I want to be included - remove #:
#  include:
#    domains:
#       - switch
#    entity_globs:
#       - sensor.power_*
#       - sensor.voltage_*
#       - sensor.current_*
#       - sensor.energy_*
#       - sensor.envoy_*
#       - sensor.growattmodbus*
#       - sensor.water*
#       - sensor.d1_02*
#       - sensor.fgms001_motion_sensor_air_temperature
#       - sensor.zw100_multisensor_6_humidity
#       - *deur*
#       - *door*
#       - *motion*
  exclude:
    domains:
      - alert
      - automation
      - weblink
      - updater
      - camera
    entities:
      - sun.sun # Don't record sun data
      - sensor.date
      - sensor.long_power_failure_count
      - sensor.power_tariff
      - sensor.short_power_failure_count
      - sensor.voltage_sags_phase_l1
      - sensor.voltage_swells_phase_l1
      - sensor.sensor_pixels
      - sensor.influxdb_db_size
      - sensor.power_kwh_to_watt
      - sensor.solar_percent_pzem 
      - sensor.pzem016modbus_a
      - sensor.pzem016modbus_w
      - sensor.solar_percent
    entity_globs:
      - sensor.binas*
      - sensor.synology*
      - sensor.double_take*
      - switch.wasmachine*
      - sensor.*_fps

I use include on an entity by entity basis. It’s a pain to maintain but Watchman helps when I forget to remove a sensor I no longer use:

host: a0d7b954-influxdb
port: 8086
database: homeassistant
username: homeassistant
password: !secret influxdb_pssword
max_retries: 3
default_measurement: state
include:
  entities:
    - binary_sensor.dark_outside
    - binary_sensor.downstairs_heat_pump_active
    - binary_sensor.front_camera_smart_motion_human
    - binary_sensor.front_camera_smart_motion_vehicle
    - binary_sensor.master_bed_occupied
    - binary_sensor.night_vision_active
    - binary_sensor.patio_camera_smart_motion_human
    - binary_sensor.pir_bathroom
    - binary_sensor.pir_dining
    - binary_sensor.pir_ensuite
    - binary_sensor.pir_entry
    - binary_sensor.pir_hall
    - binary_sensor.pir_laundry
    - binary_sensor.pir_lounge
    - binary_sensor.pir_master_bedroom
    - binary_sensor.pir_rumpus
    - binary_sensor.pir_spare_bedroom
    - binary_sensor.pir_stairwell
    - binary_sensor.pir_study
    - binary_sensor.pir_wardrobe
    - binary_sensor.pir_workshop
    - binary_sensor.rear_deck_camera_smart_motion_human
    - binary_sensor.rumpus_camera_smart_motion_human
    - binary_sensor.side_deck_camera_smart_motion_human
    - binary_sensor.side_deck_camera_smart_motion_vehicle
    - binary_sensor.spare_bed_occupied
    - binary_sensor.upstairs_heat_pump_active
    - binary_sensor.washing_camera_smart_motion_human
    - binary_sensor.workshop_camera_smart_motion_human
    - input_number.t93_daily_cost
    - input_number.master_bed_trigger_level
    - input_number.spare_bed_trigger_level
    - sensor.4g_downloaded
    - sensor.4g_uploaded
    - sensor.air_temperature
    - sensor.average_daily_cost
    - sensor.back_router_linkquality
    - sensor.bar_fridge_temperature_battery
    - sensor.bar_fridge_temperature_linkquality
    - sensor.bar_fridge_temperature_temperature
    - sensor.bathroom_dht_wifi_signal
    - sensor.bathroom_window_battery
    - sensor.bathroom_window_linkquality
    - sensor.breakfast_bar_linkquality
    - sensor.calculated_light_brightness
    - sensor.cinema_subwoofer_linkquality
    - sensor.cinema_window_battery
    - sensor.cinema_window_linkquality
    - sensor.processor_temperature
    - sensor.cumulative_energy_cost
    - sensor.database_size
    - sensor.dining_door_battery
    - sensor.dining_door_linkquality
    - sensor.dining_table_linkquality
    - sensor.dining_window_e_battery
    - sensor.dining_window_w_battery
    - sensor.dining_window_e_linkquality
    - sensor.dining_window_w_linkquality
    - sensor.dining_room_dht_wifi_signal
    - sensor.dishwasher_linkquality
    - sensor.disk_use_percent
    - sensor.doorbell_downstairs_wifi_signal
    - sensor.doorbell_upstairs_wifi_signal
    - sensor.electric_blanket_linkquality
    - sensor.electronics_bench_linkquality
    - sensor.energy_consumed_daily_total
    - sensor.energy_from_grid_daily_offpeak
    - sensor.energy_from_grid_daily_peak
    - sensor.energy_hot_water_daily_offpeak
    - sensor.energy_imported_daily_total
    - sensor.energy_inverter_daily
    - sensor.energy_to_grid_daily
    - sensor.ensuite_dht_wifi_signal
    - sensor.freezer_temperature_battery
    - sensor.freezer_temperature_linkquality
    - sensor.freezer_temperature_temperature
    - sensor.fridge_temperature_battery
    - sensor.fridge_temperature_linkquality
    - sensor.fridge_temperature_temperature
    - sensor.front_door_battery
    - sensor.front_door_linkquality
    - sensor.front_router_linkquality
    - sensor.fireplace_wifi_signal
    - sensor.grid_power
    - sensor.ha_downloaded
    - sensor.ha_uploaded
    - sensor.hobart_rain_since_9am
    - sensor.hobart_temp
    - sensor.hobart_wind_speed_knot
    - sensor.home_assistant_core_cpu_percent
    - sensor.home_assistant_core_memory_percent
    - sensor.home_assistant_supervisor_cpu_percent
    - sensor.home_assistant_supervisor_memory_percent
    - sensor.hot_water_rssi
    - sensor.hot_water_temperature
    - sensor.hourly_rainfall
    - sensor.hp_inlet_temperature
    - sensor.hp_return_temperature
    - sensor.influxdb_db_size
    - sensor.irrigation_wifi_signal
    - sensor.kitchen_leak_wifi_signal
    - sensor.kitchen_shelly_sw_rssi
    - sensor.kitchen_window_e_battery
    - sensor.kitchen_window_w_battery
    - sensor.kitchen_window_e_linkquality
    - sensor.kitchen_window_w_linkquality
    - sensor.lane_gate_battery
    - sensor.lane_gate_linkquality
    - sensor.laundry_door_battery
    - sensor.laundry_door_linkquality
    - sensor.laundry_leak_wifi_signal
    - sensor.lawn_moisture
    - sensor.led_table_linkquality
    - sensor.lounge_chair_reclined_battery
    - sensor.lounge_chair_reclined_linkquality
    - sensor.lounge_dehumidifier_control_wifi_signal
    - sensor.lounge_door_battery
    - sensor.lounge_door_linkquality
    - sensor.lounge_front_shelly_sw_rssi
    - sensor.lounge_rear_shelly_sw_rssi
    - sensor.lounge_room_dht_wifi_signal
    - sensor.lounge_room_temperature
    - sensor.lounge_window_e_battery
    - sensor.lounge_window_e_linkquality
    - sensor.lounge_window_s_battery
    - sensor.lounge_window_s_linkquality
    - sensor.mailbox_door_battery
    - sensor.mailbox_door_linkquality
    - sensor.mailbox_lid_battery
    - sensor.mailbox_lid_linkquality
    - sensor.master_bed_sensor
    - sensor.master_bedroom_dht_wifi_signal
    - sensor.master_bedroom_window_battery
    - sensor.master_bedroom_window_linkquality
    - sensor.mechanical_bench_linkquality
    - sensor.memory_use_percent
    - sensor.metering_power_absorbed
    - sensor.nas_free_pct
    - sensor.nas_mean_disk_temp
    - sensor.nbn_downloaded
    - sensor.nbn_uploaded
    - sensor.next_bill_estimate
    - sensor.nfc_scanner_wifi_signal
    - sensor.offpeak_cost_today
    - sensor.outside_light_level
    - sensor.outside_light_level_wifi_signal
    - sensor.pc_monitor_linkquality
    - sensor.peak_cost_today
    - sensor.pi3b_linkquality
    - sensor.processor_use
    - sensor.projector_uart_wifi_signal
    - sensor.pv_array_mismatch
    - sensor.pv_a_efficency
    - sensor.pv_b_efficency
    - sensor.pv_power_a
    - sensor.pv_power_b
    - sensor.rack_fan_linkquality
    - sensor.roof_space_dht_wifi_signal
    - sensor.roof_space_temperature
    - sensor.rumpus_dehumidifier_control_wifi_signal
    - sensor.rumpus_door_battery
    - sensor.rumpus_door_linkquality
    - sensor.rumpus_room_dht_wifi_signal
    - sensor.rumpus_room_temperature
    - sensor.self_solar_consumption_daily
    - sensor.shed_door_battery
    - sensor.shed_door_linkquality
    - sensor.shelly_em_rssi
    - sensor.soil_saturation
    - sensor.solar_earnings_today
    - sensor.solar_forecast_error
    - sensor.solar_power_consumed
    - sensor.spare_bed_sensor
    - sensor.spare_bedroom_dht_wifi_signal
    - sensor.spare_bedroom_window_battery
    - sensor.spare_bedroom_window_linkquality
    - sensor.spb_electric_blanket_linkquality
    - sensor.study_window_n_battery
    - sensor.study_window_n_linkquality
    - sensor.study_window_s_battery
    - sensor.study_window_s_linkquality
    - sensor.tank_temperature
    - sensor.tempered_water_temperature
    - sensor.toilet_window_battery
    - sensor.toilet_window_linkquality
    - sensor.total_cost_today
    - sensor.total_grid_power
    - sensor.total_power_consumed
    - sensor.uv_index
    - sensor.upstairs_heat_pump_power
    - sensor.washing_machine_linkquality
    - sensor.wled_wi_fi_rssi
    - sensor.workshop_chargers_linkquality
    - sensor.workshop_switch_wifi_signal
    - sensor.workshop_window_battery
    - sensor.workshop_window_linkquality
    - switch.hot_water
    - switch.zone_1
    - switch.zone_2
    - switch.zone_3

I keep 2 years of data and the size is at 4.5GB and falling as I removed some chatty wind speed sensors 2 years ago.

:rofl:

Why not use entity_globs for e.g. pir or battery?

And why include link quality etc.?

For precisely the reason this topic was written. The include logic is obscure at best, broken at worst.

Not that it is any of your business, it is so I can monitor long term trends after adding new repeater devices.

1 Like

I meanwhile stopped forwarding data to influx( I never really had a good case for analysing older data), thus my configuration might no longer work. also never had Influx2 running

I created template sensors for all entities I wanted to have in influx, all template sensors had a time based attribute to force updates every 5 minutes and all template sensors had a “_db” at the end of the entity name ( and a few where HA added the “_2” suffix…

My Influx config looked like this:

influxdb:
  host: 127.0.0.1
  port: 8086
  include:
    entity_globs:
      - binary_sensor.*_db
      - sensor.*_db
      - sensor.*_db_2

And I had all these template sensors excluded from the standard recorder because I didn’t want to store the data twice:

recorder:
  exclude:
    entity_globs:
      - binary_sensor.*_db
      - sensor.*_db
      - sensor.*_db_2

Additionally, because I didn’t like that by default the measurement in influx was created by the unit of the entity ( measurements like “°C” or “%”), I overwrote the measurements in the influx configuration:

influxdb:
  include:
    entity_globs:
        ...
  component_config_glob:
    sensor.*humidity*:
      override_measurement: humidity
    sensor.*temperature*:
      override_measurement: temperature
    sensor.*valve*:
      override_measurement: valve
    sensor.*battery*:
      override_measurement: battery
    sensor.*akkufullstand*:
      override_measurement: akku
    sensor.*climate*:
      override_measurement: climate
    sensor.*synology*:
      override_measurement: synology

Armin

1 Like

I am sorry. I asked for maybe a reason I would need it as well.

I started another home assistant to “test”…

This is so true! Even if I specify specific entities (two) though everything is written in the DB…

According to the documentation this should not happen but it does…
With that in mind it’s not even feasible to start changing the config in my production setup…

This was my test config:

influxdb:
    include:
        entity_globs:
          - sensor.p1*
          - sensor.solar*

Then when alle entities entered influx I tried:

influxdb:
    include:
        entity_globs:
          - sensor.p1*
          - sensor.solar*
    exclude:
        entity_globs:
          - sensor.experia*

But that did not change anything…

I do not understand these filters.
Why is everything included by default?

I only want to track a couple of dedicated entities and nothing else but I am not able to configure this. The influxdb is always tracking tons of useless information and start growing very fast.

It would be great if some one could provide a example that allows to pick only a few entities and remove the rest.

If you only want to track a few entities the best way is to do this:

influxdb:
  host: a0d7b954-influxdb
  port: 8086
  database: homeassistant
  username: homeassistant
  password: !secret influxdb_pssword
  max_retries: 3
  default_measurement: state
  include:
    entities:
      - binary_sensor.dark_outside # your list of entities here
      - binary_sensor.deck_rear_smart_motion
      - binary_sensor.deck_side_smart_motion

I was searching something else and found this post. It’s a matter of fact that HA writes the sensor data every 30? seconds independent if value changed or not. This blows up the database, not limited to influxdb, same for logger-db.

A question if somebody has an idea how to delete sequent updates whithout value update what are more or less duplicates.

Sample:
01.06.2024 08:00:30 - temp - 24
01.06.2024 08:01:00 - temp - 24 <<<< duplicate
01.06.2024 08:01:30 - temp - 24 <<<< duplicate
01.06.2024 08:02:00 - temp - 25
01.06.2024 08:02:30 - temp - 25 <<<< duplicate

How could be functional “<<<< duplicate” removed from database or even prevent to be written? When temp wasn’t changed there is no benefit to write or keep such entries.

No it is not a fact at all. The data is only written when the state changes. This actually annoys a lot of people. See: InfluxDB to store data in intervals

Is that Influxdb or the recorder database?

If it is the recorder, maybe it is because one of the entity’s attributes changes every 30 seconds.

Ok, when data is only stored when it’s unique it’s the behaviour I would appriciate. Probably there are some special situations where this is not the case. I think it’s what you meant in your last sentence … if one of the attributes changed … e. g. climate has many attributes so it’s likely that there was any change. Out of the discussion I assume I can create / use sensors with single state and then there should be no duplicates.

Data posted was simplified to show the relevant part. I extracted a measurement from my Influxdb and there are duplicates (same values in a sequence) still. See below.

Sample …

state,domain=climate,entity_id=hmburo temperature=11 1718339860104157000
state,domain=climate,entity_id=hmburo temperature=11 1718341379117318000
state,domain=climate,entity_id=hmburo temperature=11 1718341554125495000
state,domain=climate,entity_id=hmburo temperature=11 1718341714743475000
state,domain=climate,entity_id=hmburo temperature=11 1718341860911716000
state,domain=climate,entity_id=hmburo temperature=11 1718342631170018000
state,domain=climate,entity_id=hmburo temperature=11 1718343230934009000
state,domain=climate,entity_id=hmburo temperature=11 1718343540399376000
state,domain=climate,entity_id=hmburo temperature=11 1718343856145836000
state,domain=climate,entity_id=hmburo temperature=11 1718344285407929000
state,domain=climate,entity_id=hmburo temperature=11 1718344712941242000
state,domain=climate,entity_id=hmburo temperature=11 1718345337215762000
state,domain=climate,entity_id=hmburo temperature=11 1718347095981940000