🛡 Watchman - keeps track of missing entities and actions in your config files

poudenes · January 24, 2023, 5:56pm

Hi All,

Im using this integration many many times. Will it even be able to create a “Watchman unused” version?
A new integration that checks if entities created by users like sensors, binary_sensors, groups, manual lights, switches… all domains… not used in automations anymore.

Sometimes I check al my created things (testing here testing there, try to create something nice, but not work) and then you have leftovers that not used anymore.

Would be so nice to see a list of all domain entities that are not used. So you can remove them easier…

Cheers

davinci · January 28, 2023, 9:15am

I changed some settings to exclude custom_components. After a restart I got this error:

The system cannot restart because the configuration is not valid: Invalid config for [watchman]: expected a dictionary for dictionary value @ data['watchman']. Got 'w'. (See ?, line ?).

How can I fix this? Where is this invalid config? I can’t configure from the UI anymore.

parautenbach · January 28, 2023, 12:59pm

Lets start with your config.

davinci · January 28, 2023, 1:29pm

Sure, but I have set all the configs in the UI and since the custom component won’t load I can’t check for that.

parautenbach · January 28, 2023, 1:34pm

Do you have watchman: as a key in any of your YAML files?

davinci · January 28, 2023, 1:44pm

Yes. I just checked it and it was watchman: w. How embarrassing…
Of course without w it works.

Thanks for pointing me in the right direction.

Haze · February 4, 2023, 9:17pm

I am trying to get the markdown cards working.
I was able to get the missing entities showing up, but the missing services is not.
I am using the code as is:

- type: markdown
            content: >-
              <h2 class="some"> <ha-icon icon='mdi:shield-half-full'></ha-icon> Watchman report</h2>
              <h3> Missing Services: {{ states.sensor.watchman_missing_services.state }} </h3>
              {%- for item in state_attr("sensor.watchman_missing_services", "services") %}
              <hr><table><tr> <td>  <ha-icon icon='mdi:cloud-alert'></ha-icon> {{ item.id }}
              <a title="{{item.occurrences}}">{{item.occurrences.split('/')[-1].split(':')[0]}}</a>
              </td></tr></table>
              {%- endfor %}
            card_mod:
              style:
                ha-markdown:
                  $: |
                    ha-markdown-element:first-of-type hr{
                      border-color: #303030;
                    }

The spaces for some reason get screwed up when I paste from yaml to the forum…so ignore the extra spaces seen here.
If I only put

<h2 class="some"> <ha-icon icon='mdi:shield-half-full'></ha-icon> Watchman report</h2>
<h3> Missing Services: {{ states.sensor.watchman_missing_services.state }} </h3>

It shows 1 service missing.

So I put the actual template generating the list into Dev → Template, and I get "TypeError: 'NoneType' object is not iterable"

Any clues?

poudenes · February 9, 2023, 9:37am

Additional maybe new feature:

Let watchman create a list of all entities that are used in automations.
This idea popped up because is was checking my MariaDB and it was 2.9GB

Im not collect manual all entities that needs history etc. But would be much easier to see a full list just like this:

<domain>.<entitiy_id>
<domain>.<entitiy_id>
<domain>.<entitiy_id>
<domain>.<entitiy_id>

Then poeple can easily update or change their recorder: and history: part.

tom_l · February 9, 2023, 10:21am

Run this in the template editor:

=========================
{% for state in states %}
  - {{ state.entity_id -}}
{% endfor %}

poudenes · February 9, 2023, 2:13pm

This give me all entities in a list. Not the one that are used in an automation. But with this list it make it already some easier to handle

dgcd · February 14, 2023, 7:48pm

Hi everyone…

I am really very new in HA and I add this capability into my system but I receive into report this messag

Congratulations, all 13 entities from your config are available!

The real true is that system has two entities in unavailable state…can you please help me to understand what is happening here?

All the best!

HaraldGithub · February 21, 2023, 9:01pm

Hi,
very good tool, I like it very much (Great work by you!).
Have you think about checking the node-red-flows, too? I’m using only NR for all my automations and therefor it would be very helpful if there is a NR-watchman-checker on board.

OzGav · February 21, 2023, 10:58pm

Is there anything in the config ignored options? How many entities are shown if you do what Tom suggests four posts above?

PlayFaster · February 22, 2023, 2:07am

Hi There

This works for me, after some modification, to display missing services. Breaking out all of the lines is not required, it just helped me for clarity.

Markdown to display WatchMan report missing services …

  - type: markdown
    content: >-
      <h2>
      <ha-icon icon='mdi:shield-half-full'>
      </ha-icon>
      Services report
      </h2>
      <h3> Missing Services:
      {{ states.sensor.watchman_missing_services.state }}
      </h3>
            {%- for item in state_attr("sensor.watchman_missing_services","entities") %}
      <hr>
      <table>
      <tr>
      <td>
      {{ item.id }}
 
      <a title="{{item.occurrences}}">
      {{item.occurrences.split('/')[-1] }}
      </a>
      </td>
      </tr>
      </table>
      {%- endfor %}
    card_mod:
      style:
        ha-markdown:
          $: |
            ha-markdown-element:first-of-type hr{
              border-color: #303030;
            }

Haze · February 22, 2023, 10:31pm

This worked or me. Thank you!

TRusselo · March 2, 2023, 5:16pm

I have added a few things to make my Watchman Card more helpful to me:
Tracking the number of ZWave Devices alive vs. connected.
Tracking the number of Zigbee2MQTT devices.

sensor.yaml

  - platform: template
    sensors:
       count_zwave_devices:
        friendly_name: 'ZWave'
        icon_template: mdi:z-wave
        value_template: >
          {% set total = states|selectattr('object_id', 'search', 'node_status')|list|count %}
          {% set dead = states|selectattr('object_id', 'search', 'node_status')|selectattr('state', 'in', 'dead, unavailable, unknown') | list | count %}
          {{ total - dead }} Alive/{{ total }} Total

      count_zigbee_devices:
        friendly_name: 'Zigbee'
        icon_template: mdi:z-wave
        value_template: >
          {% set entities = integration_entities('mqtt') %}
          {% set ns = namespace(unique_ids = []) %}
          {% for entity in entities %}
            {% if device_id(entity) not in ns.unique_ids %}
              {% set ns.unique_ids = ns.unique_ids + [device_id(entity)] %}
            {% endif%}
          {% endfor %}
          {{ ns.unique_ids | count -2 }}  ## -2 was added for my config to be accurate, yours will differ. i have 2 mqtt devices that are not zigbee

Watchman Report Card:

<h2> <ha-icon icon='mdi:shield-half-full'></ha-icon> Watchman report</h2> <h3>Missing Entities: {{ states.sensor.watchman_missing_entities.state }} </h3> {%- for item in state_attr("sensor.watchman_missing_entities", "entities") %} <hr> <table><tr> <td> <ha-icon icon='mdi: {%- if item.state=="missing"-%}cloud-alert' {%- elif item.state=="unavail" -%}cloud-off-outline' {%- else-%}cloud-question' {%- endif -%} ></ha-icon> {{ item.id }} [{{item.state}}] <a title="{{item.occurrences}}"> {{item.occurrences.split('/')[-1].split(':')[0]}}</a> </td></tr></table> {%- endfor %}
<h2> <ha-icon icon='mdi:z-wave'></ha-icon> ZWave report</h2> <h3>Devices: {{ states.sensor.count_zwave_devices.state }}
<h2> <ha-icon icon='mdi:zigbee'></ha-icon> Zigbee report</h2> <h3>Devices: {{ states.sensor.count_zigbee_devices.state }}

Ronnie-J · March 9, 2023, 1:41pm

Does anyone know why I keep getting an error trying to get Watchman to send me a telegram notification?

Trying via the developer panel caling the report service;

service: watchman.report
data:
  create_file: true
  send_notification: true
  service: notify.telegram
  data:
    parse_mode: html
  parse_config: true
  chunk_size: true

Logger: homeassistant.helpers.script.websocket_api_script
Source: custom_components/watchman/__init__.py:422
Integration: Watchman ([documentation](https://github.com/dummylabs/thewatchman), [issues](https://github.com/dummylabs/thewatchman/issues))
First occurred: 2:37:59 PM (2 occurrences)
Last logged: 2:38:31 PM

websocket_api script: Error executing script. Unexpected error for call_service at pos 1: the JSON object must be str, bytes or bytearray, not dict

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 451, in _async_step await getattr(self, handler)() File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 684, in _async_call_service_step await service_task File "/usr/src/homeassistant/homeassistant/core.py", line 1808, in async_call task.result() File "/usr/src/homeassistant/homeassistant/core.py", line 1845, in _execute_service await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)( File "/config/custom_components/watchman/__init__.py", line 240, in async_handle_report await async_report_to_notification( File "/config/custom_components/watchman/__init__.py", line 422, in async_report_to_notification data = {} if service_data is None else json.loads(service_data) File "/usr/local/lib/python3.10/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not dict

kbrown01 · March 14, 2023, 6:53pm

If anyone would like a Mushroom-ized version of this, I created one in my install:

type: custom:stack-in-card
title: Possible Issues
cards:
  - type: custom:auto-entities
    card:
      type: grid
      columns: 1
      square: false
    card_param: cards
    filter:
      template: >-
        {%- for item in
        state_attr("sensor.watchman_missing_entities","entities") -%}
          {{
            {
              'type': "custom:mushroom-template-card",
              'entity': item.id,
              'primary': item.state + ' ' + item.id,
              'secondary': item.occurrences,
              'icon_color': "red" if item.state == "missing" else "orange" if item.state == "unavail" else "yellow",
              'icon': "mdi:cloud-alert" if item.state == "missing" else "mdi:cloud-off-outline" if item.state == "unavail" else "mdi:cloud-question"
              }
            }},
        {%- endfor -%}

Looks like this:

teskanoo · April 2, 2023, 10:28pm

I have managed to achieve this - a bit late but I hope it helps someone


# https://github.com/custom-cards/flex-table-card
- type: custom:flex-table-card
  title: 'Watchman Missing Entities'
  entities:
    include: sensor.watchman_missing_entities
  columns:
  - name: Entity
    align: left
    data: entities
    modify: x.id
  - name: State
    align: left
    data: entities
    modify: x.state
  - name: Occurrences
    align: left
    data: entities
    modify: x.occurrences

chinedu40 · April 4, 2023, 12:34pm

Is there a way to set up an automation that run only when one specific entity is marked as unavailable or missing? I’ve tried using the following as a conditional template “{{ ‘camera.netatmo_camera’ in state_attr(‘sensor.watchman_missing_entities’, ‘entities’) }}” but it is always false event when camera.netatmo_camera is listed under ID in state attributes