hmmm, thats what I did.
btw, what does ./py-spy do? I never used that so maybe thats the source for issues…
also, these files now reside in the /config folder. Is that where they are supposed to be?
When you run a program without a path, $PATH is searched for the program. When you specify a path with ./ (aka current directory) your shell runs the program at that path
yes! that works!
right, so what do you need me to spy on exactly. if I enter ./py-spy top, it errors with no required --pid
so how do I find the correct pid for the instance you need
any of these (top in a normal terminal window) ?
ive just given that a shot using the 554 pid, and yet it returns:
even though I am doing what the link says to do:
You can also use py-spy from the Host OS to profile a running process running inside the docker container.
Or? am I in the wrong directory
Ive even tried to add –cap-add SYS_PTRACE to the ini_commands in the add-on options, but that results in a fatal issue…
wait… something is happening… I redid the whole procedure after deleting the earlier files, and now I can run the top on home assistant
hope this means I can now finally do what you need me to…pfff
so, what do you need. 
did a first run on homeassistant and dropped the SVG in your DB address (cant show it here apparently)
The svg showed a lot (that’s a bit of an understatement) of yaml loading overhead which that YAML pr that already merged should make a lot better once we switch over to python 3.10 which has the new wheels with the native code. Hopefully this week
The other big hitter that was obvious so far is the json decode overhead which is addressed by the orjson PR.
That’s all I could see on my phone. I’ll take a better look through it when I get home in a few hours
1 Like
To do a deeper analysis I’ll need to get the following py-spys
dump
top run for 5m
A recording for 2m with rate 100
A recording for 5m with rate 200
great!
I’ll try and figure out the correct commands for that, and see how I can copy from that ssh window, which up to now it wont allow me to do. maybe I can dump output to a file too. Ill check the syntax
thanks!
Thanks, these are some great insights
I’m surprised just how much of your profiles are YAML and JSON overhead. The PRs I previously linked should take a big chunk out of the YAML and JSON overhead.
The only other thing that stands out is template overhead but I’m not sure we can do much about that. It’s below both the YAML and JSON overhead on the profiles though; so even if we can’t do much for that, addressing the serializer overhead is going to be a big win.
1 Like
which app do you use for the .svg files? I have Inkscape opening them, but dont think that optimal…
Open them in chrome.
A few more insights from your profiles
General YAML parsing overhead - already addressed via https://github.com/home-assistant/core/pull/73337 but not yet effective until we switch to python 3.10. 
HACS - JSON encoding overhead (addressed via https://github.com/home-assistant/core/pull/72847)
Unifi - JSON decoding overhead (would need a separate change to aiounifi after https://github.com/home-assistant/core/pull/72847 – https://github.com/Kane610/aiounifi/pull/145)
Websocket - JSON endcoding overhead (addressed via https://github.com/home-assistant/core/pull/72847)
Storage writes - JSON encoding overhead (addressed via https://github.com/home-assistant/core/pull/72847)
Google Assistant generating the supported entities list iterates the entire state machine. However this might only happen very infrequently since it’s a point in time snapshot. Thats probably responsible for quite a bit of your cpu time. This approach was likely fine when there were < 200 entities in most instance’s state machines, but once things got to scale its starting to be a problem.
Template overhead is a bit high, but you have a lot of templates so thats likely not much that can be done there.
One more I missed in the profile because its small, but luftdaten is creating a new httpx instance each update which also may leak memory according to this github issue Memory leak when creating lots of AsyncClient contexts · Issue #978 · encode/httpx · GitHub
There may be a reason it works this way though so its not something we could change without the codeowner’s input (It could simplify be that nobody has gotten around to using the helper though)
We have a helper to get a shared client here core/httpx_client.py at a92105171cc69a5ada661a67a3abfaf8fa55b213 · home-assistant/core · GitHub
Cool, thats completely new to me. Magic how that allows to see the underlying processes. thx.
And those are some amazing findings, and glad you’re already working on the fixes.
besides the templating, there’s not much fro the user to do here apparently?
Wondering if fixing these will make the Rpi4 back a preferred device too, since everybody kept saying to me it was the hardware that caused this, and why dont I use a heavier pc…
Ill leave my Pi4 config in its current state, and continue for now on my Mini PC (which made these py-spy recordings), also because it is much more power efficient 
For the Luftdaten (which is a bad integration in itself, many of the endpoints are very unreliable, I have been wondering to uninstall ): should I file an issue to get this tracked? I did file Luftdaten: map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ConnectTimeout · Issue #61687 · home-assistant/core · GitHub a long time ago and that too mentions the httpx, albeit in a connection timeout
Id be happy to provide new recordings without the Luftdaten, if you would think that to be useful, (and proof for the inefficiency in the readings)
On the Google assistant:
why would it iterate the full state machine, when there is an explicit Include list available in the
cloud:
google_actions:
filter:
include_entities:
configuration?
on that last subject: I figure the same rules for include/entities apply compared to recorder?
eg:
if I only explicitly include a few switches, it leaves the others out? Or, do I need to exclude the domain switch to leave them all out, and include a few individual switches I need.
just a thought:
on the Yaml, could this be a lead:
I have my dashboards configured in
lovelace:
mode: yaml
resources: !include /config/dashboard/resources.yaml
dashboards:
ui-overview:
mode: yaml
filename: dashboard/dashboards/ui-overview.yaml
title: Overview
icon: mdi:view-dashboard
ui-settings:
mode: yaml
filename: dashboard/dashboards/ui-settings.yaml
title: Change settings
icon: mdi:tools
ui-data:
mode: yaml
filename: dashboard/dashboards/ui-data.yaml
title: Home data
icon: mdi:home-analytics
require_admin: true
ui-develop:
mode: yaml
filename: dashboard/dashboards/ui-develop.yaml
title: Develop
icon: mdi:remote-desktop
require_admin: true
ui-test:
mode: yaml
filename: dashboard/dashboards/ui-test.yaml
title: Test
icon: mdi:test-tube
require_admin: true
and not in the /config folder.
Ive filed an issue for that some time ago but no response was ever received unfortunately
if Frontend keeps looking for that, it might take a while
It won’t make much difference right now since py-spy does sampling. All the JSON overhead time is so much of the profile that we won’t be able to see some of the smaller branches until that is sorted out.
Since it is a filter it needs to check everything against the filter to see if it matches. If you only have include_entities, it is just one type of filter. It could be optimized to only use the included entities instead but since there are ~9 filter cases it would only make it faster for that one specific case. We may be able to re-use some of the performance improvements we did in emulated_hue since its a similar performance issue.
I was able to squeeze a bit more (~19%) performance out of the templates since some of the overhead is wrapping states with template objects Speed up generation of template states by bdraco · Pull Request #73728 · home-assistant/core · GitHub
sure. on those templates:
I did change several bigger templates with more than a few templates attributes from verbose to using the this variable. eg
- unique_id: composite_marijn_location
<<: &composite_location
name: >
{% set id = this.attributes.id|default('Not yet set') %}
{{id|title}} location
state: >
{% set id = this.attributes.id|default('Not yet set') %}
{{states('device_tracker.'~id~'_app')|replace('_',' ')|title}}
picture: >
{% set id = this.attributes.id|default('Not yet set') %}
/local/family/{{id}}_home.png
attributes:
id: marijn
<<: &composite_attributes
location: >
{% set id = this.attributes.id|default('Not yet set') %}
{% set sensor = 'sensor.'~id~'_app_geocoded_location' %}
{{state_attr(sensor,'Name')}} {{state_attr(sensor,'Postal Code')}} {{state_attr(sensor,'Locality')}} {{state_attr(sensor,'Country')}}
country_code: >
{% set id = this.attributes.id|default('Not yet set') %}
{{state_attr('sensor.'~id~'_app_geocoded_location','ISO Country Code')|lower}}
country: >
{% set id = this.attributes.id|default('Not yet set') %}
{{state_attr('sensor.'~id~'_app_geocoded_location','Country')}}
latitude: >
{% set id = this.attributes.id|default('Not yet set') %}
{% set track = 'device_tracker.'~id~'_app' %}
{{state_attr(track,'latitude')}}
longitude: >
{% set id = this.attributes.id|default('Not yet set') %}
{% set track = 'device_tracker.'~id~'_app' %}
{{state_attr(track,'longitude')}}
and reuse those with
- unique_id: composite_wijke_location
<<: *composite_location
attributes:
id: wijke
<<: *composite_attributes
etc etc
several other cases too, so, question is if that this variable, though nice to write in YAML, doesn’t harm the efficiency?
At the time of migration I did get the feeling it was upping the startup times, but thats only a feeling , not anything scientific. let alone a py-spy recording…
o, I’ve been trying to copy a Dump reading in the SSH window, but am not able to copy anything in those SSH windows inside the HA instance at all. (either community or official for that matter)
How can I get you a read out other than a screenshot ? I am out of ideas…
If you want standard ssh access you can get into the host with
Same docker command to get into the container
Made it in, py-spy dump and top coming up anytime now
reason I was asking is because next the that single include for include_entities I exclude all others with
exclude_domains:
- alarm_control_panel
- binary_sensor
- button
- camera
# - climate
- cover
- fan
- group
- input_boolean
- input_select #(helper)
- light
# - lock
- media_player
- scene
- script
- sensor
- switch
- vacuum
- select
reason I commented the domains here, is because of the included entities of those domains. Maybe thats not even necessary if include_entities would really only include those, and nothing else.
I am still not sure of the logic here, because I did some experimenting, and have several cover entities in the included_entities listed, and yet excluded the domains for those. eg cover/switch/script.
It actually doesn’t add the other covers (and other domains) that are also available in my system. But it didn’t do that without the exclude_domains either…
So, I guess I can delete the whole exclude_domains section?
If only for clarity, but also for not forcing the system to check those filters…?
Seeing the google_assistant total percentage in Top, it might well be worth trying to get that as efficient as possible indeed.
update
not sure what happened but it must have been either updating to todays dev nightly 2022.7.0.dev20220621, or deleting the exclude_domains from cloud: but my startup has improved a lot. So much that I couldn’t believe it, and Ive even rebooted the host to make it run a full startup. Listing has changed dramatically:
and should be compared with the listing in How to install Py spy on a HA OS instance, please instruct - #10 by Mariusthvdb
From your top:
The google assistant issue is not infrequent as I had hoped. Its pretty clear its the number 2 cpu/runtime time usage past JSON.
10.00% 10.00% 5.88s 5.93s supported (homeassistant/components/google_assistant/trait.py)
10.00% 20.00% 2.96s 9.00s <listcomp> (homeassistant/components/google_assistant/helpers.py
YAML is a close 3rd but thats just waiting on the new python version (really the JSON PR is as well)
Sadly we are back to me not having a google assistant so there isn’t much I can do for that.