How can I run the built in HA benchmark script on HassIO?

I’m trying to run the build in benchmark script on HassIO as detailed here:

Has anyone managed to do it? If so how?

yep

(homeassistant) homeassistant@nuccy:/home/nick$ hass --script benchmark async_million_events
Using event loop: asyncio.unix_events
Benchmark async_million_events done in 0.7540765689918771s
Benchmark async_million_events done in 0.7610460779978894s

Although I am not using hassio.

Try this:

#Open HA container shell
docker container exec -it homeassistant bash
#Run Benchmark
hass --script benchmark async_million_events

Using event loop: uvloop
Benchmark async_million_events done in 6.156998317019315s
Benchmark async_million_events done in 6.122647846990731s

I’m using HassIO on a RPi3 so that would be why I’m almost ten times slower than @nickrout’s nuc :slight_smile: However I’m not sure if it’s working 100% because the event loop is different.

2 Likes

That worked after disabling protection mode. Hassio on a Chinese mini PC:
Annotation%202019-08-19%20222402

Thanks @HypnoToad those commands seemed to have worked for me.

Right now I’m testing 2 x RPi 3’s, one running HassIO and the other running Hass with VENV. Both Pi’s are using the same SD card type, and official PSU, the only difference here is software.

Believe it or not, the HassIO benchmarks are quite a bit better despite Docker being used.

This is what I get with VENV:

Using event loop: asyncio.unix_events
Benchmark async_million_events done in 9.03224296999997s

And this is what I get with HassIO:

Using event loop: uvloop
Benchmark async_million_events done in 6.146387876000954s

It seems that using uvloop may give a 50% boost in network performance in this benchmark at least. It’s a shame there aren’t more benchmarks really, I’d quite like to try a file IO comparison.

1 Like

Does anyone know how to run the benchmark test when Home Assistant runs in a Docker container (not hass.io). It doesn’t appear to contain the “hass” command.

I’m using Portainer, I opened a console window in the Home Assistant container, ran the following command:

find / -name "hass"

and it came back empty-handed. Without it, I don’t see how it’s possible to run the benchmark test …

Screenshot%20from%202019-08-20%2022-03-51

I guess you could download the script and run it locally?

https://github.com/jest-community/jest-pytest/blob/b197b0b31e3ca5c411202d97583cbd2d2b0b92e9/src/tests/integration/home-assistant/homeassistant/scripts/benchmark/init.py

I thought hass was the home assistant run file.

I think the script is present in the Docker container:

Screenshot%20from%202019-08-21%2008-10-13

Me too which is why it’s perplexing that hass can’t be found in the container.

I peeked into /etc/systemd/system, hoping to find some clue of how Home Assistant is started, but found nothing related to it. :man_shrugging:

What comes up if you do

find / | grep "/hass"

In the docker container?

It returns more results than fit in the console’s buffer … because there are several directories that begin with /hass, each containing many files.

Added a | more to the command but there are still too many results to step through …

Hmm, Ok actually if nothing came up when you did

find / -name "hass"

then there probably isn’t a file called ‘hass’ anywhere, very odd.

Just to be sure, when HA us running, try

ps -A | grep -i hass

and see if you get any output

This isn’t the result I expected:

root@plutonium:/usr/src/app# ps -A
  PID TTY          TIME CMD
    1 ?        00:00:11 python
   58 pts/0    00:00:00 bash
   73 pts/0    00:00:00 ps
root@plutonium:/usr/src/app#

Perhaps my docker-fu is weak but I did expect to see a few more processes running within the container …

I’m not that experienced either I’m afraid, give ps -aux a try, it may show more info. :slight_smile: pstree is nice also, if it’s installed that is.

pstree isn’t installed in the container but ps -aux reveals a little bit more information:

root@plutonium:/usr/src/app# ps -aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  5.1  2.4 1324768 96268 ?       Ssl  08:11   0:11 python -m homeassistant --config /config
root        57  0.2  0.0   5752  3604 pts/0    Ss   08:11   0:00 bash
root        64  0.0  0.0   9392  2988 pts/0    R+   08:14   0:00 ps -aux

It would appear that Home Assistant is initiated without the use of hass.

python -m homeassistant --config /config

Maybe

python -m homeassistant --script benchmark async_million_events

PS my hass script, on my venv system is pretty short

#!/srv/homeassistant/bin/python3
# -*- coding: utf-8 -*-
import re
import sys

from homeassistant.__main__ import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

The ‘-m’ is telling python to run the homeassistant library as a script.

you may just be able to run the benchmark script directly with python, something like

python /path/to/benchmark/__init__.py async_million_events

may work

Some progress but not the desired result just yet:

root@plutonium:/usr/src/app# python /usr/src/app/homeassistant/scripts/benchmark/__init__.py async_million_events
Traceback (most recent call last):
  File "/usr/src/app/homeassistant/scripts/benchmark/__init__.py", line 10, in <module>
    from homeassistant import core
ModuleNotFoundError: No module named 'homeassistant'

The tenth line in the benchmark script attempts to get a module from somewhere it can’t find.

I’m guessing the hass script sets up the correct environment so that when the benchmark script is called it finds all required modules.

My production system is running Home Assistant in a python virtual environment. I’ll take a peak at its copy of hass and see what it does.

HA 2021.1.2
Supervised on Ubuntu 18.04.5

bash-5.0# hass --script benchmark async_million_events

I’v got:

usage: hass [-h] [--script {benchmark}]
            {fire_events,time_changed_helper,state_changed_helper,state_changed_event_helper,logbook_filtering_state,logbook_filtering_attributes,_logbook_filtering,filtering_entity_id,valid_entity_id,json_serialize_states}
hass: error: argument name: invalid choice: 'async_million_events' (choose from 'fire_events', 'time_changed_helper', 'state_changed_helper', 'state_changed_event_helper', 'logbook_filtering_state', 'logbook_filtering_attributes', '_logbook_filtering', 'filtering_entity_id', 'valid_entity_id', 'json_serialize_states')