HA 0.116 / Supervisor 247 - Core and Supervisor & Addon Stats

[silvrr]

With the release of v0.116 (Really Supervisor v247) we are provided some usage stats about the core and supervisor. These are useful but I’m guessing some people want to see this information on the frontend or have some historical data available. Some quick use of the developer tools in Chrome and some probing of the rest API and we can pull this info into a rest sensor.

For examples pulling indivdiual stats about an add-ons usage see this post.

First we need to pull the data from the Rest API.

#https://www.home-assistant.io/integrations/rest/
  - platform: rest
    resource: http://192.168.1.24:8123/api/hassio/core/stats
    name: core
    unit_of_measurement: '%'
    value_template: '{{ value_json.data.cpu_percent }}'
    scan_interval: 10
    headers:
      Authorization: !secret llt
      Content-Type: application/json
    json_attributes_path: "$.data"
    json_attributes:
      - memory_percent

  - platform: rest
    resource: http://192.168.1.24:8123/api/hassio/supervisor/stats
    name: supervisor
    unit_of_measurement: '%'
    value_template: '{{ value_json.data.cpu_percent }}'
    scan_interval: 10
    headers:
      Authorization: !secret llt
      Content-Type: application/json
    json_attributes_path: "$.data"
    json_attributes:
      - memory_percent

Information on the Rest API can be found HERE and information on the restful sensor is included in the YAML snippet above.

You can adjust the scan_interval to suit your needs, however, be aware that a very short scan interval will require frequent API calls. Likely not a issue but if your network/hardware is stressed this may cause issues.

!secret llt is a long lived token token in the format
llt: "Bearer eyJ0eXAiOiJKV1QiLCJhSUPERLONgTOKENlZWQ0NzBhYLe8sxN8nup1Uw"

sensor.core and sensor.supervisor will report the CPU percent and have attributes available with the other sensors as an attribute(s). You can make another sensor the primary or remove some of the unused attributes if you want.

You can add other attributes from the API by adding the value as a json_attribute restful sensor. The raw output of the API looks as follows:

{
"result": "ok",
"data":{
"cpu_percent": 0.02,
"memory_usage": 72126464,
"memory_limit": 4028276736,
"memory_percent": 1.79,
"network_rx": 16357595,
"network_tx": 19696684,
"blk_read": 10268672,
"blk_write": 233472
}
}

To get the attributes into a sensor usable in the frontend or in your history, just grab the attribute into a template sensor. Examples below from each of the sensors.

#https://www.home-assistant.io/integrations/template/
  - platform: template
    sensors:
      core_mem:
        friendly_name: Core Memory Usage
        unit_of_measurement: '%'
        value_template: "{{ state_attr('sensor.core', 'memory_percent') }}"

  - platform: template
    sensors:
      supervisor_mem:
        friendly_name: Supervisor Memory Usage
        unit_of_measurement: '%'
        value_template: "{{ state_attr('sensor.supervisor', 'memory_percent') }}"

Example integration into the frontend:

[tom_l]

@silvrr

I’m guessing some people want to see this information on the frontend or have some historical data available.

You guessed correctly.

Thank you for this. Will give it a go later.

The core stats are available from the system monitor integration though. It’s only the supervisor stats I was after.

[silvrr]

@tom_l

The core stats are available from the system monitor integration though

I think the system monitor integration is different than just the core stats. See my core memory usage vs. the ‘total system memory usage’ in my example screenshot. The system monitor seems to be pulling in add-on memory usage. This is my assumption at least, given that if I stop an add-on that uses a lot of memory (e.g. Unifi Controller) that sensor drops significantly while core stays the same.

[tom_l]

Yeah that makes sense.

Why is the core cpu use greater than the total though?

[silvrr]

@tom_l

Why is the core cpu use greater than the total though?

I would assume it comes down to timing. I have my rest sensors on a 10 sec interval and Im not sure what the frequency is for the system_monitor integration.

[MrBorg (P. Borg)]

So what you are saying is that this was easier before (pre 0.116) to add system info to front-end?

Only these lines before:

And now a bit more complex with the REST part and Templates etc.?

[silvrr]

No, this is pulling different information. The systemmonitor integration is still available in 0.116.0.

When running HassOS you are running a Operating System and a number of Docker containers ontop of that OS. Two of those containers are the Core Container and the Supervisor Container.

These sensors pull the usage from the individual containers. The system monitor integration you screenshotted pulls the stats from the host (the metrics the Operating System is seeing) The core sensors are pulling the usage metrics for only the core container. Same for the supervisor sensors.

Hope that helps.

[MrBorg (P. Borg)]

Ok that’s a relief but has something changed in the systemmonitor part?

I now get this error:

And I get no system figures beside temperature:

[silvrr]

@MrBorg

has something changed in the systemmonitor part?

Yes, always read the breaking changes before updating.

0.116: Entities card row editor, restore snapshots and performance metrics

[MrBorg (P. Borg)]

You are so right but I actually did read it and from the first part here I could see that I should not worry:

But figuring out what the Log is saying is not that easy (not for me and maybe others)

[MrBorg (P. Borg)]

[silvrr]

For network in and network out you need to provide the name of the interface to monitor. See the docs for an example.

System Monitor

[Markus99 (Mark)]

Exactly, the update now requires some type elements to also have arg. On my NUC I use:

  - type: throughput_network_in
    arg: eno1
  - type: throughput_network_out
    arg: eno1

Docs updated above show which are now required.

[SylvainGa (Sylvain Gagnon)]

Can’t get it to work sensor.core says unavailable. This is what shows up in the log, every 30 seconds (polling time)

2020-10-11 21:42:52 WARNING (SyncWorker_27) [homeassistant.components.rest.sensor] Empty reply found when expecting JSON data

This is the config under sensor: It’s basically a copy/paste of the first post config text but with my own internal IP (http i working) and replaced llt by respapi which is defined in secret.yaml with the corresponding long lived token.

  - platform: rest
    resource: http://192.168.2.84:8123/api/hassio/core/stats
    name: core
    unit_of_measurement: '%'
    value_template: '{{ value_json.data.cpu_percent }}'
    scan_interval: 30
    headers:
      Authorization: !secret restapi
      Content-Type: application/json
    json_attributes_path: "$.data"
    json_attributes:
      - memory_percent

  - platform: rest
    resource: http://192.168.2.84:8123/api/hassio/supervisor/stats
    name: supervisor
    unit_of_measurement: '%'
    value_template: '{{ value_json.data.cpu_percent }}'
    scan_interval: 30
    headers:
      Authorization: !secret restapi
      Content-Type: application/json
    json_attributes_path: "$.data"
    json_attributes:
      - memory_percent

  - platform: template
    sensors:
      core_mem:
        friendly_name: Core Memory Usage
        unit_of_measurement: '%'
        value_template: "{{ state_attr('sensor.core', 'memory_percent') }}"

  - platform: template
    sensors:
      supervisor_mem:
        friendly_name: Supervisor Memory Usage
        unit_of_measurement: '%'
        value_template: "{{ state_attr('sensor.supervisor', 'memory_percent') }}"

Don’t know what I’m doing wrong.

[tom_l]

Does your token secret have this form?

 "Bearer eyJ0eXAiOiJKV1QiLCJhSUPERLONgTOKENlZWQ0NzBhYLe8sxN8nup1Uw"

i.e. does it have the word “Bearer” in front of the token?

[silvrr]

Ok, one more addition to this thread. After seeing another post (sorry I lost it and can’t link it) I discovered that you can pull the same usage information for each of your addons.

Example for my unifi controller addon.

  - platform: rest
    resource: http://192.168.1.24:8123/api/hassio/addons/a0d7b954_unifi/stats
    name: unifi
    unit_of_measurement: '%'
    value_template: '{{ value_json.data.cpu_percent }}'
    scan_interval: 30
    headers:
      Authorization: !secret restapi
      Content-Type: application/json
    json_attributes_path: "$.data"
    json_attributes:
      - memory_percent

The output of this api endpoint is the same as the core and supervisor.

{
"result": "ok",
"data":{
"cpu_percent": 0.52,
"memory_usage": 639205376,
"memory_limit": 3133292544,
"memory_percent": 20.4,
"network_rx": 278650723,
"network_tx": 1107050279,
"blk_read": 149344256,
"blk_write": 289079296
}
}

[tom_l]

YES!

I have been chasing down an intermittent (but prolonged - requiring a restart to stop) high core CPU use for weeks. Profiler and py-spy have failed to reveal the culprit. So my next option was going to be disabling addons one by one.

I’ll implement these sensors instead and see if that helps track the issue down. Thank you.

EDIT: haven’t seen one of these for a while:

All appears to be working though:

[tom_l]

Well that was a dead end. Though I am beginning to suspect something is amiss with the api call for the core cpu %. This:

- platform: rest
  resource: https://mydomain_redacted.duckdns.org/api/hassio/core/stats
  name: Core CPU
  unit_of_measurement: '%'
  value_template: '{{ value_json.data.cpu_percent }}'
  scan_interval: 30
  headers:
    Authorization: !secret ha_api_token
    Content-Type: application/json
  json_attributes_path: "$.data"
  json_attributes:
    - memory_percent

Is currently showing:

And as you can see from the graph it has been growing steadily up.

However, monitoring the same statistic in the supervisor it never gets above 7%:

EDIT: I just saw the supervisor core cpu monitor hit 98%, very briefly - about the same time as the api sensor updated. Is the cpu API call causing the cpu use to skyrocket?

Why does restarting home assistant stop this happening?

EDIT2: in case this is caused by multiple api calls happening at once I have updated my rest sensors to use different prime number scan intervals, from 23 to 71 seconds.

[tom_l]

One more update. After speaking with Ludeeus on Discord, rather than use the external API access like:

  resource: https://mydomain_redacted.duckdns.org/api/hassio/core/stats

If you use internal access to the API this considerably reduces the overhead. As I use duckDNS and its security certs I have to use https and so accessing

    resource: http://supervisor/core/stats

will not work for me. If you can access home assistant locally via http use that resource. However if you use TLS certs directly in home assistant you can can get around this by using a command line sensor instead:

- platform: command_line
  name: Core CPU
  command: 'curl -sSL -H "Authorization: Bearer $SUPERVISOR_TOKEN" http://supervisor/core/stats'
  unit_of_measurement: '%'
  value_template: "{{ (value_json.data.cpu_percent|float/4)|round(2) }}"
  scan_interval: 60
  json_attributes: 
    - data

- platform: command_line
  name: Supervisor CPU
  command: 'curl -sSL -H "Authorization: Bearer $SUPERVISOR_TOKEN" http://supervisor/supervisor/stats'
  unit_of_measurement: '%'
  value_template: "{{ (value_json.data.cpu_percent|float/4)|round(2) }}"
  scan_interval: 60
  json_attributes: 
    - data

- platform: template
  sensors:
    core_memory:
      friendly_name: "Core RAM"
      value_template:  "{{ (state_attr('sensor.core_cpu', 'data')|to_json|from_json).memory_percent }}"
      unit_of_measurement: '%'
    supervisor_memory:
      friendly_name: "Supervisor RAM"
      value_template: "{{ (state_attr('sensor.supervisor_cpu', 'data')|to_json|from_json).memory_percent }}"
      unit_of_measurement: '%'

Note that as I am using a four thread CPU the percent CPU value has to be divided by 4. Adjust the template for your hardware.

Finally an addon example for completeness:

- platform: command_line
  name: Grafana CPU
  command: 'curl -sSL -H "Authorization: Bearer $SUPERVISOR_TOKEN" http://supervisor/addons/a0d7b954_grafana/stats'
  unit_of_measurement: '%'
  value_template: "{{ (value_json.data.cpu_percent|float/4)|round(2) }}"
  scan_interval: 60
  json_attributes: 
    - data

- platform: template
  sensors:
    grafana_memory:
      friendly_name: "Grafana RAM"
      value_template: "{{ (state_attr('sensor.grafana_cpu', 'data')|to_json|from_json).memory_percent }}"
      unit_of_measurement: '%'

This has considerably reduced the CPU % when

[Mariusthvdb (Marius)]

as I am on Homeassistant OS and always use the !secret for the bearer token, please le me ask where your token is stored when you use "Authorization: Bearer $SUPERVISOR_TOKEN" ?

or in other words, can I rewrite this:

  - platform: rest
    name: Hassio Rpi4 config
    resource: !secret resource_hassio_rpi4_config
    value_template: > #components|list|count}}
      {{value_json.version}}
    json_attributes:
      - components
      - unit_system
      - config_dir
    headers:
      Content-Type: application/json
      Authorization: !secret api_bearer_token
      User-Agent: Home Assistant REST sensor
    scan_interval: 3600

to the sensors you use with the ‘http://supervisor/core/stats’ as resource?