InfluxDB - Change exclude to Include - Examples

Ok, so I wanted to put some effort in my ever growing influxdb which is now around 40Gb in size :-).

But like any other “tasks” in HA it’s not so simple.

I read about influxdb and I use the addon. I Assume I have v1 (not 2).
I have and exclude list now in recorder and in influxbd, mostly the same.
I tried to search how to cleanup influx, not found any helpful sources.
Then I tried to find out how to get “wanted” entities in influx:

  • only certain integrations (cannot find how to do this).

All these tasks… I end up in dead ends…

Thus I wanted to ask, just as a starter, can anyone who has an “include” list for influx please share it below?

Additionally - is it a good moment to start influxdb v2 ? Can 1 and 2 run in parallel?

This is my current config and added “include” config.
I am struggling to understand the rules of InfluxDB - Home Assistant

I expect that:

  1. the include will be included, then nothing else will be included.
  2. the exclude will all be extra excluded on top of the nothing else will be included.
influxdb:
# below the newly added entities I want to be included - remove #:
#  include:
#    domains:
#       - switch
#    entity_globs:
#       - sensor.power_*
#       - sensor.voltage_*
#       - sensor.current_*
#       - sensor.energy_*
#       - sensor.envoy_*
#       - sensor.growattmodbus*
#       - sensor.water*
#       - sensor.d1_02*
#       - sensor.fgms001_motion_sensor_air_temperature
#       - sensor.zw100_multisensor_6_humidity
#       - *deur*
#       - *door*
#       - *motion*
  exclude:
    domains:
      - alert
      - automation
      - weblink
      - updater
      - camera
    entities:
      - sun.sun # Don't record sun data
      - sensor.date
      - sensor.long_power_failure_count
      - sensor.power_tariff
      - sensor.short_power_failure_count
      - sensor.voltage_sags_phase_l1
      - sensor.voltage_swells_phase_l1
      - sensor.sensor_pixels
      - sensor.influxdb_db_size
      - sensor.power_kwh_to_watt
      - sensor.solar_percent_pzem 
      - sensor.pzem016modbus_a
      - sensor.pzem016modbus_w
      - sensor.solar_percent
    entity_globs:
      - sensor.binas*
      - sensor.synology*
      - sensor.double_take*
      - switch.wasmachine*
      - sensor.*_fps

I use include on an entity by entity basis. It’s a pain to maintain but Watchman helps when I forget to remove a sensor I no longer use:

host: a0d7b954-influxdb
port: 8086
database: homeassistant
username: homeassistant
password: !secret influxdb_pssword
max_retries: 3
default_measurement: state
include:
  entities:
    - binary_sensor.dark_outside
    - binary_sensor.downstairs_heat_pump_active
    - binary_sensor.front_camera_smart_motion_human
    - binary_sensor.front_camera_smart_motion_vehicle
    - binary_sensor.master_bed_occupied
    - binary_sensor.night_vision_active
    - binary_sensor.patio_camera_smart_motion_human
    - binary_sensor.pir_bathroom
    - binary_sensor.pir_dining
    - binary_sensor.pir_ensuite
    - binary_sensor.pir_entry
    - binary_sensor.pir_hall
    - binary_sensor.pir_laundry
    - binary_sensor.pir_lounge
    - binary_sensor.pir_master_bedroom
    - binary_sensor.pir_rumpus
    - binary_sensor.pir_spare_bedroom
    - binary_sensor.pir_stairwell
    - binary_sensor.pir_study
    - binary_sensor.pir_wardrobe
    - binary_sensor.pir_workshop
    - binary_sensor.rear_deck_camera_smart_motion_human
    - binary_sensor.rumpus_camera_smart_motion_human
    - binary_sensor.side_deck_camera_smart_motion_human
    - binary_sensor.side_deck_camera_smart_motion_vehicle
    - binary_sensor.spare_bed_occupied
    - binary_sensor.upstairs_heat_pump_active
    - binary_sensor.washing_camera_smart_motion_human
    - binary_sensor.workshop_camera_smart_motion_human
    - input_number.t93_daily_cost
    - input_number.master_bed_trigger_level
    - input_number.spare_bed_trigger_level
    - sensor.4g_downloaded
    - sensor.4g_uploaded
    - sensor.air_temperature
    - sensor.average_daily_cost
    - sensor.back_router_linkquality
    - sensor.bar_fridge_temperature_battery
    - sensor.bar_fridge_temperature_linkquality
    - sensor.bar_fridge_temperature_temperature
    - sensor.bathroom_dht_wifi_signal
    - sensor.bathroom_window_battery
    - sensor.bathroom_window_linkquality
    - sensor.breakfast_bar_linkquality
    - sensor.calculated_light_brightness
    - sensor.cinema_subwoofer_linkquality
    - sensor.cinema_window_battery
    - sensor.cinema_window_linkquality
    - sensor.processor_temperature
    - sensor.cumulative_energy_cost
    - sensor.database_size
    - sensor.dining_door_battery
    - sensor.dining_door_linkquality
    - sensor.dining_table_linkquality
    - sensor.dining_window_e_battery
    - sensor.dining_window_w_battery
    - sensor.dining_window_e_linkquality
    - sensor.dining_window_w_linkquality
    - sensor.dining_room_dht_wifi_signal
    - sensor.dishwasher_linkquality
    - sensor.disk_use_percent
    - sensor.doorbell_downstairs_wifi_signal
    - sensor.doorbell_upstairs_wifi_signal
    - sensor.electric_blanket_linkquality
    - sensor.electronics_bench_linkquality
    - sensor.energy_consumed_daily_total
    - sensor.energy_from_grid_daily_offpeak
    - sensor.energy_from_grid_daily_peak
    - sensor.energy_hot_water_daily_offpeak
    - sensor.energy_imported_daily_total
    - sensor.energy_inverter_daily
    - sensor.energy_to_grid_daily
    - sensor.ensuite_dht_wifi_signal
    - sensor.freezer_temperature_battery
    - sensor.freezer_temperature_linkquality
    - sensor.freezer_temperature_temperature
    - sensor.fridge_temperature_battery
    - sensor.fridge_temperature_linkquality
    - sensor.fridge_temperature_temperature
    - sensor.front_door_battery
    - sensor.front_door_linkquality
    - sensor.front_router_linkquality
    - sensor.fireplace_wifi_signal
    - sensor.grid_power
    - sensor.ha_downloaded
    - sensor.ha_uploaded
    - sensor.hobart_rain_since_9am
    - sensor.hobart_temp
    - sensor.hobart_wind_speed_knot
    - sensor.home_assistant_core_cpu_percent
    - sensor.home_assistant_core_memory_percent
    - sensor.home_assistant_supervisor_cpu_percent
    - sensor.home_assistant_supervisor_memory_percent
    - sensor.hot_water_rssi
    - sensor.hot_water_temperature
    - sensor.hourly_rainfall
    - sensor.hp_inlet_temperature
    - sensor.hp_return_temperature
    - sensor.influxdb_db_size
    - sensor.irrigation_wifi_signal
    - sensor.kitchen_leak_wifi_signal
    - sensor.kitchen_shelly_sw_rssi
    - sensor.kitchen_window_e_battery
    - sensor.kitchen_window_w_battery
    - sensor.kitchen_window_e_linkquality
    - sensor.kitchen_window_w_linkquality
    - sensor.lane_gate_battery
    - sensor.lane_gate_linkquality
    - sensor.laundry_door_battery
    - sensor.laundry_door_linkquality
    - sensor.laundry_leak_wifi_signal
    - sensor.lawn_moisture
    - sensor.led_table_linkquality
    - sensor.lounge_chair_reclined_battery
    - sensor.lounge_chair_reclined_linkquality
    - sensor.lounge_dehumidifier_control_wifi_signal
    - sensor.lounge_door_battery
    - sensor.lounge_door_linkquality
    - sensor.lounge_front_shelly_sw_rssi
    - sensor.lounge_rear_shelly_sw_rssi
    - sensor.lounge_room_dht_wifi_signal
    - sensor.lounge_room_temperature
    - sensor.lounge_window_e_battery
    - sensor.lounge_window_e_linkquality
    - sensor.lounge_window_s_battery
    - sensor.lounge_window_s_linkquality
    - sensor.mailbox_door_battery
    - sensor.mailbox_door_linkquality
    - sensor.mailbox_lid_battery
    - sensor.mailbox_lid_linkquality
    - sensor.master_bed_sensor
    - sensor.master_bedroom_dht_wifi_signal
    - sensor.master_bedroom_window_battery
    - sensor.master_bedroom_window_linkquality
    - sensor.mechanical_bench_linkquality
    - sensor.memory_use_percent
    - sensor.metering_power_absorbed
    - sensor.nas_free_pct
    - sensor.nas_mean_disk_temp
    - sensor.nbn_downloaded
    - sensor.nbn_uploaded
    - sensor.next_bill_estimate
    - sensor.nfc_scanner_wifi_signal
    - sensor.offpeak_cost_today
    - sensor.outside_light_level
    - sensor.outside_light_level_wifi_signal
    - sensor.pc_monitor_linkquality
    - sensor.peak_cost_today
    - sensor.pi3b_linkquality
    - sensor.processor_use
    - sensor.projector_uart_wifi_signal
    - sensor.pv_array_mismatch
    - sensor.pv_a_efficency
    - sensor.pv_b_efficency
    - sensor.pv_power_a
    - sensor.pv_power_b
    - sensor.rack_fan_linkquality
    - sensor.roof_space_dht_wifi_signal
    - sensor.roof_space_temperature
    - sensor.rumpus_dehumidifier_control_wifi_signal
    - sensor.rumpus_door_battery
    - sensor.rumpus_door_linkquality
    - sensor.rumpus_room_dht_wifi_signal
    - sensor.rumpus_room_temperature
    - sensor.self_solar_consumption_daily
    - sensor.shed_door_battery
    - sensor.shed_door_linkquality
    - sensor.shelly_em_rssi
    - sensor.soil_saturation
    - sensor.solar_earnings_today
    - sensor.solar_forecast_error
    - sensor.solar_power_consumed
    - sensor.spare_bed_sensor
    - sensor.spare_bedroom_dht_wifi_signal
    - sensor.spare_bedroom_window_battery
    - sensor.spare_bedroom_window_linkquality
    - sensor.spb_electric_blanket_linkquality
    - sensor.study_window_n_battery
    - sensor.study_window_n_linkquality
    - sensor.study_window_s_battery
    - sensor.study_window_s_linkquality
    - sensor.tank_temperature
    - sensor.tempered_water_temperature
    - sensor.toilet_window_battery
    - sensor.toilet_window_linkquality
    - sensor.total_cost_today
    - sensor.total_grid_power
    - sensor.total_power_consumed
    - sensor.uv_index
    - sensor.upstairs_heat_pump_power
    - sensor.washing_machine_linkquality
    - sensor.wled_wi_fi_rssi
    - sensor.workshop_chargers_linkquality
    - sensor.workshop_switch_wifi_signal
    - sensor.workshop_window_battery
    - sensor.workshop_window_linkquality
    - switch.hot_water
    - switch.zone_1
    - switch.zone_2
    - switch.zone_3

I keep 2 years of data and the size is at 4.5GB and falling as I removed some chatty wind speed sensors 2 years ago.

:rofl:

Why not use entity_globs for e.g. pir or battery?

And why include link quality etc.?

For precisely the reason this topic was written. The include logic is obscure at best, broken at worst.

Not that it is any of your business, it is so I can monitor long term trends after adding new repeater devices.

1 Like

I meanwhile stopped forwarding data to influx( I never really had a good case for analysing older data), thus my configuration might no longer work. also never had Influx2 running

I created template sensors for all entities I wanted to have in influx, all template sensors had a time based attribute to force updates every 5 minutes and all template sensors had a “_db” at the end of the entity name ( and a few where HA added the “_2” suffix…

My Influx config looked like this:

influxdb:
  host: 127.0.0.1
  port: 8086
  include:
    entity_globs:
      - binary_sensor.*_db
      - sensor.*_db
      - sensor.*_db_2

And I had all these template sensors excluded from the standard recorder because I didn’t want to store the data twice:

recorder:
  exclude:
    entity_globs:
      - binary_sensor.*_db
      - sensor.*_db
      - sensor.*_db_2

Additionally, because I didn’t like that by default the measurement in influx was created by the unit of the entity ( measurements like “°C” or “%”), I overwrote the measurements in the influx configuration:

influxdb:
  include:
    entity_globs:
        ...
  component_config_glob:
    sensor.*humidity*:
      override_measurement: humidity
    sensor.*temperature*:
      override_measurement: temperature
    sensor.*valve*:
      override_measurement: valve
    sensor.*battery*:
      override_measurement: battery
    sensor.*akkufullstand*:
      override_measurement: akku
    sensor.*climate*:
      override_measurement: climate
    sensor.*synology*:
      override_measurement: synology

Armin

1 Like

I am sorry. I asked for maybe a reason I would need it as well.

I started another home assistant to “test”…

This is so true! Even if I specify specific entities (two) though everything is written in the DB…

According to the documentation this should not happen but it does…
With that in mind it’s not even feasible to start changing the config in my production setup…

This was my test config:

influxdb:
    include:
        entity_globs:
          - sensor.p1*
          - sensor.solar*

Then when alle entities entered influx I tried:

influxdb:
    include:
        entity_globs:
          - sensor.p1*
          - sensor.solar*
    exclude:
        entity_globs:
          - sensor.experia*

But that did not change anything…

I do not understand these filters.
Why is everything included by default?

I only want to track a couple of dedicated entities and nothing else but I am not able to configure this. The influxdb is always tracking tons of useless information and start growing very fast.

It would be great if some one could provide a example that allows to pick only a few entities and remove the rest.

If you only want to track a few entities the best way is to do this:

influxdb:
  host: a0d7b954-influxdb
  port: 8086
  database: homeassistant
  username: homeassistant
  password: !secret influxdb_pssword
  max_retries: 3
  default_measurement: state
  include:
    entities:
      - binary_sensor.dark_outside # your list of entities here
      - binary_sensor.deck_rear_smart_motion
      - binary_sensor.deck_side_smart_motion

I was searching something else and found this post. It’s a matter of fact that HA writes the sensor data every 30? seconds independent if value changed or not. This blows up the database, not limited to influxdb, same for logger-db.

A question if somebody has an idea how to delete sequent updates whithout value update what are more or less duplicates.

Sample:
01.06.2024 08:00:30 - temp - 24
01.06.2024 08:01:00 - temp - 24 <<<< duplicate
01.06.2024 08:01:30 - temp - 24 <<<< duplicate
01.06.2024 08:02:00 - temp - 25
01.06.2024 08:02:30 - temp - 25 <<<< duplicate

How could be functional “<<<< duplicate” removed from database or even prevent to be written? When temp wasn’t changed there is no benefit to write or keep such entries.

No it is not a fact at all. The data is only written when the state changes. This actually annoys a lot of people. See: InfluxDB to store data in intervals

Is that Influxdb or the recorder database?

If it is the recorder, maybe it is because one of the entity’s attributes changes every 30 seconds.

Ok, when data is only stored when it’s unique it’s the behaviour I would appriciate. Probably there are some special situations where this is not the case. I think it’s what you meant in your last sentence … if one of the attributes changed … e. g. climate has many attributes so it’s likely that there was any change. Out of the discussion I assume I can create / use sensors with single state and then there should be no duplicates.

Data posted was simplified to show the relevant part. I extracted a measurement from my Influxdb and there are duplicates (same values in a sequence) still. See below.

Sample …

state,domain=climate,entity_id=hmburo temperature=11 1718339860104157000
state,domain=climate,entity_id=hmburo temperature=11 1718341379117318000
state,domain=climate,entity_id=hmburo temperature=11 1718341554125495000
state,domain=climate,entity_id=hmburo temperature=11 1718341714743475000
state,domain=climate,entity_id=hmburo temperature=11 1718341860911716000
state,domain=climate,entity_id=hmburo temperature=11 1718342631170018000
state,domain=climate,entity_id=hmburo temperature=11 1718343230934009000
state,domain=climate,entity_id=hmburo temperature=11 1718343540399376000
state,domain=climate,entity_id=hmburo temperature=11 1718343856145836000
state,domain=climate,entity_id=hmburo temperature=11 1718344285407929000
state,domain=climate,entity_id=hmburo temperature=11 1718344712941242000
state,domain=climate,entity_id=hmburo temperature=11 1718345337215762000
state,domain=climate,entity_id=hmburo temperature=11 1718347095981940000

hey @tom_l
May I ask you as expert a question
I would like to work with INCLUDE as only option
anything that is not defined in Include should not be persisted in influx

in this include I would need to define something with a wildcard or integration level

like

include:
  entities:
    - anything from Integration XYZ (has unfortunetly no good name pattern in it I could use)
    - any sensor value that has "kwh" in its name somewhere  - could define it *kwh* or is wildcard only as ending allowed?

thjose 2 examples are a little painful to me to understand and I would appreciate some hints.

Thank you!

EDIT:
I went through all my entities and this would be the list I want to INCLUDE and nothing else

espicially the * wildcards I used below would be interesting if this is a correct syntax?

Include:
sensor.*fronius*
sensor.local_ecowitt*
climate.*
sensor.energy*
sensor.*_battery   
sensor.*_battery_level
sensor.*_illuminance
sensor.*_temperature
sensor.*_humidity
sensor.*_smoke
sensor.*_smoke_concentration
sensor.*_energy
sensor.*_power
sensor.*_current
sensor.*_kwh
sensor.*_w
sensor.qemu*
sensor.*proxmox*
sensor.wpl17*
sensor.custom*
vacuum.greta
vacuum.roborock_s7

Exclude:
sensor.custom_ui_grp_motion

ok if anybody is looking for more examples.
I use this now and when I check in Grafana some example entities it looks like its working and the pattern with * are accepted this way

  include:                    # Beginne die Liste der zu speichernden Entities
    entities:                # Liste der spezifischen Entities ohne Wildcards
      - 'vacuum.greta'      # Der Staubsauger mit dem Namen 'greta'
      - 'vacuum.roborock_s7' # Der Staubsauger Roborock S7
      - 'vacuum.gustav'     # Der Staubsauger mit dem Namen 'gustav'

    entity_globs:           # Liste der Entities, die mit Wildcards erfasst werden
      - 'sensor.local_ecowitt*'               # Alle Entities, die mit 'sensor.local_ecowitt' beginnen
      - 'climate.*'                            # Alle Klimageräte (z.B. Thermostate)
      - 'sensor.energy*'                       # Alle Energiesensoren (z.B. Verbrauch)
      - 'sensor.*_battery'                     # Alle Batteriestatus-Entities
      - 'sensor.*_battery_level'               # Alle Batterielevel-Entities
      - 'sensor.*_illuminance'                 # Alle Helligkeitssensoren
      - 'sensor.*_temperature'                  # Alle Temperatursensoren
      - 'sensor.*_humidity'                     # Alle Feuchtigkeitssensoren
      - 'sensor.*_smoke'                        # Alle Rauchsensoren
      - 'sensor.*_smoke_concentration'         # Alle Rauchkonzentrationssensoren
      - 'sensor.*_energy'                       # Alle Energiemessungen
      - 'sensor.*_power'                        # Alle Leistungsmessungen
      - 'sensor.*_current'                      # Alle Strommessungen
      - 'sensor.*_kwh'                          # Alle kWh-Messungen
      - 'sensor.*_w'                            # Alle Watt-Messungen
      - 'sensor.qemu*'                          # Alle QEMU-bezogenen Sensoren
      - 'sensor.*proxmox*'                      # Alle Proxmox-bezogenen Sensoren
      - 'sensor.wpl17*'                         # Alle WPL17-bezogenen Sensoren
      - 'sensor.custom*'                        # Alle benutzerdefinierten Sensoren

Ok, bit the bullet and changed my exclude to include… but this does not work like expected (or I may not know what I am doing).

I have:

include:
    entities:
        - sensor.fgms001_motion_sensor_air_temperature
        - binary_sensor.blauwe_hek
        - sensor.zw100_multisensor_6_humidity
    domains:
        - switch
        - valve
    entity_globs:
        - sensor.current_*
        - sensor.d1_02*
        - sensor.energy_*
        - sensor.envoy_*
        - sensor.growattmodbus*
        - sensor.power_*
        - sensor.solar*
        - sensor.voltage_*
        - sensor.water*
        - .*deur*
        - .*door*
        - .*motion*
        - .*temp*

But I keep seeing entities being recreated in the influxdb (I use InfluxDB Studio)…