How to throttle Supervisor's version check?

123 · March 2, 2021, 10:28pm

The Supervisor checks for new version information by accessing version.home-assistant.io every 10 minutes. My pihole log shows this behavior (repeating every ten minutes) where two requests are normally made for each poll.

Show pihole log

I have two instances of Home Assistant and I don’t need or want either of them to poll so frequently.

As far as I know, there’s no provision to permit a user to adjust the polling frequency. So I looked at Supervisor’s code to see if I could change its 10-minute interval to 60 minutes (i.e. modify its code). I believe this is the portion that accesses and interprets the version data:
supervisor/updater.py at main · home-assistant/supervisor · GitHub

However, it doesn’t appear to contain any reference to a 10-minute time interval so I assume something else controls the polling frequency. Does anyone know how it works and where one would adjust its polling interval?

Burningstone · March 3, 2021, 6:55am

Hi Taras, did you check the tasks.py already?

github.com

home-assistant/supervisor/blob/7e94537e36c820896ad3f3738cf99c3035b035d0/supervisor/misc/tasks.py

"""A collection of tasks."""
import logging

from ..const import AddonState, HostFeature
from ..coresys import CoreSysAttributes
from ..exceptions import (
    AddonsError,
    AudioError,
    CliError,
    CoreDNSError,
    HomeAssistantError,
    MulticastError,
    ObserverError,
)
from ..jobs.decorator import Job, JobCondition

_LOGGER: logging.Logger = logging.getLogger(__name__)

HASS_WATCHDOG_API = "HASS_WATCHDOG_API"

This file has been truncated. show original

123 · March 3, 2021, 2:54pm

Thanks for taking the time to investigate.

I’ve reviewed it and I can’t find anything that schedules the updater to run every 10 minutes (600 seconds).

For example, this constant (tasks.py, line 38) sets an interval of 120 minutes:

RUN_RELOAD_UPDATER = 7200

It’s used here (tasks.py, line 74):

self.sys_scheduler.register_task(self.sys_updater.reload, RUN_RELOAD_UPDATER)

From what I can understand, it registers a task to repeat at an interval set by RUN_RELOAD_UPDATER, namely every 120 minutes. The task is to execute self.sys_updater.reload.

Here is the reload method for the Updater class:

github.com

home-assistant/supervisor/blob/7e94537e36c820896ad3f3738cf99c3035b035d0/supervisor/updater.py#L49


def __init__(self, coresys):
    """Initialize updater."""
    super().__init__(FILE_HASSIO_UPDATER, SCHEMA_UPDATER_CONFIG)
    self.coresys = coresys

async def load(self) -> None:
    """Update internal data."""
    with suppress(HassioUpdaterError):
        await self.fetch_data()

async def reload(self) -> None:
    """Update internal data."""
    with suppress(HassioUpdaterError):
        await self.fetch_data()

@property
def version_homeassistant(self) -> Optional[str]:
    """Return latest version of Home Assistant."""
    return self._data.get(ATTR_HOMEASSISTANT)

@property

The important bit is that on line 52 it calls fetch_data() which is where it retrieves data from version.home-assistant.io.

Long story short, it appears to schedule a task to reload the version data every 120 minutes. However, pihole’s logs are showing it happens every 10 minutes (600 seconds), not every 120 minutes (7200 seconds).

I can’t make the connection between what is actually happening and what the code says will happen.

CentralCommand · March 3, 2021, 4:29pm

That is… extremely confusing. I can’t make heads or tails of it either, it definitely looks like its schedule is set to 120 minutes.

However if your goal is to modify the code you are running locally and not to submit a PR adding a new option for people to use I think there is a way. Right before the offending fetch_data method there is an annotation which throttles it:

github.com

home-assistant/supervisor/blob/455bbc457be0d5c8fd0ebc4cdc78ec1e48ba1d3d/supervisor/updater.py#L173



@channel.setter
def channel(self, value: UpdateChannel):
    """Set upstream mode."""
    self._data[ATTR_CHANNEL] = value

@Job(
    conditions=[JobCondition.INTERNET_SYSTEM],
    on_condition=UpdaterJobError,
    limit=JobExecutionLimit.THROTTLE_WAIT,
    throttle_period=timedelta(seconds=30),
)
async def fetch_data(self):
    """Fetch current versions from Github.

    Is a coroutine.
    """
    url = URL_HASSIO_VERSION.format(channel=self.channel)
    machine = self.sys_machine or "default"

    try:

As you can see, that’s set to prevent it from running more then once every 30 seconds currently. If you bumped that timedelta up to one hour I think it would prevent it from running more then once an hour regardless of what was actually doing the scheduling.

I can’t say for sure that wouldn’t break other things but seems worth a try.

[EDIT] You might also have to investigate what that limit=JobExecutionLimit.THROTTLE_WAIT means. That sounds a little bit like it would queue up requests if something was sending requests too frequently given the throttle period which would be bad. Ideally you would want it to reject requests during the throttle period not queue them until the hour had passed. That enum is defined here but the options aren’t explained so might require some trial and error.

ludeeus · March 3, 2021, 4:33pm

It’s not version check, it’s online check

123 · March 3, 2021, 4:36pm

Can you elaborate on what you mean by “not version check, it’s online check”?

fetch_data() appears to be the only place that references version.home-assistant.io. If I’m wrong about that, can you help me understand what exactly is making Supervisor check that URL every 10 minutes?

I’d like to increase the interval to 1 hour or more.

ludeeus · March 3, 2021, 4:40pm

github.com

home-assistant/supervisor/blob/169c7ec00425de29d6f81f6d9c4a66e03c1a685f/supervisor/supervisor.py#L198


    try:
        await self.instance.retag()
    except DockerError:
        _LOGGER.error("Repair of Supervisor failed")

async def check_connectivity(self):
    """Check the connection."""
    timeout = aiohttp.ClientTimeout(total=10)
    try:
        await self.sys_websession.head(
            "https://version.home-assistant.io/online.txt", timeout=timeout
        )
    except (ClientError, asyncio.TimeoutError):
        self._connectivity = False
    else:
        self._connectivity = True

123 · March 3, 2021, 4:48pm

Thank you! So in addition to a task that retrieves the latest version data (every 120 minutes) there’s a task that performs a connectivity check (and both tasks use the same base URL).

The only fly in the ointment is that the connectivity check’s polling interval doesn’t appear to match the observed interval:

github.com

home-assistant/supervisor/blob/169c7ec00425de29d6f81f6d9c4a66e03c1a685f/supervisor/misc/tasks.py#L50


RUN_WATCHDOG_CLI_DOCKER = 60
RUN_WATCHDOG_OBSERVER_DOCKER = 60
RUN_WATCHDOG_MULTICAST_DOCKER = 60

RUN_WATCHDOG_ADDON_DOCKER = 30
RUN_WATCHDOG_ADDON_APPLICATON = 120
RUN_WATCHDOG_OBSERVER_APPLICATION = 180

RUN_REFRESH_ADDON = 15

RUN_CHECK_CONNECTIVITY = 30


class Tasks(CoreSysAttributes):
    """Handle Tasks inside Supervisor."""

    def __init__(self, coresys):
        """Initialize Tasks."""
        self.coresys = coresys
        self._cache = {}

        # Connectivity
        self.sys_scheduler.register_task(
            self._check_connectivity, RUN_CHECK_CONNECTIVITY
        )

I assumed all polling intervals were expressed in seconds but that would imply a 30-second interval! Even if it were in minutes it still doesn’t jibe with the observed 10-minute interval.

There’s still a missing piece to this puzzle.

ludeeus · March 3, 2021, 4:53pm

It’s throttled in the function in the bottom of that file if internet is available

123 · March 3, 2021, 4:59pm

Yeah, my mistake, I didn’t examine check_connectivity's code.

It explicitly mentions it performs a full version check “each 10 min”. I assume the 30-second interval represents the ‘clock tick’ used to execute the task but the task itself will poll the URL at a different interval.

github.com

home-assistant/supervisor/blob/169c7ec00425de29d6f81f6d9c4a66e03c1a685f/supervisor/misc/tasks.py#L455


        if addon.in_progress or await addon.is_running():
            continue

        # Adjust state
        addon.state = AddonState.STOPPED

async def _check_connectivity(self) -> None:
    """Check system connectivity."""
    value = self._cache.get("connectivity", 0)

    # Need only full check if not connected or each 10min
    if value >= 600:
        pass
    elif (
        self.sys_supervisor.connectivity
        and self.sys_host.network.connectivity is None
    ) or (
        self.sys_supervisor.connectivity
        and self.sys_host.network.connectivity is not None
        and self.sys_host.network.connectivity
    ):

Thanks again! I need some time to understand what it’s doing exactly before I attempt to modify it.

ludeeus · March 3, 2021, 5:02pm

Don’t modify it;

It’s lost on updates
several functions inside the supervisor blindly rely on that functioning as written

123 · March 3, 2021, 5:19pm

I have a test and a production system. I intend to modify the code used in the test system and observe the results. If there are no ill effects over several days then I’ll consider modifying the production system.

Thanks for the warning about losing changes on updates. I’m aware of this because I run a patch on the homekit_controller integration every time I update the production system.

Modify an integration’s source code inside a docker container.

FWIW, the integration reports light level as a float which results in generating more data than I find to be necessary. The patch simply truncates the value to int. Practically speaking, I’ve found no need for a light level of 165.23 vs simply 165.

123 · March 7, 2021, 3:00pm

UPDATE

I’ve increased the connectivity check’s polling interval to 2 hours with a full version-check occurring every 24 hours. Nothing seems to have been negatively affected. For convenience, I’ve created a shell script to ‘patch’ the code because, as mentioned, the modifications are undone when Supervisor is updated.

That all works fine on my production system running Home Assistant Supervised on Debian. However, my test system runs Home Assistant OS and it performs two independent connectivity checks.

Supervisor checks connectivity (in the same way as in Home Assistant Supervised)
The OS itself performs a connectivity check every 5 minutes (via NetworkManager).

github.com

home-assistant/operating-system/blob/e646650a7ee2b43827be4ee9b377d336e74feb18/buildroot-external/rootfs-overlay/etc/NetworkManager/NetworkManager.conf#L17


[keyfile]
unmanaged-devices=type:bridge;type:tun;type:veth

[logging]
backend=journal

[connection]
connection.mdns=2
connection.llmnr=2

[connectivity]
uri=https://version.home-assistant.io/online.txt

[device]
wifi.scan-rand-mac-address=no

I’ve tried to modify NetworkManager.conf, following the documentation’s instructions, but it seems to reject all modifications to connectivity. I’ve posted an Issue in the Operating System repo to determine if this is by design or a bug.

tl;dr
Currently, I am unable to prevent Home Assistant OS from performing an Internet connectivity check every 5 minutes. In contrast, I’ve had success with Home Assistant Supervised.

123 · May 8, 2021, 4:07pm

UPDATE

Sadly, my simple modification of Supervisor’s source-code is no longer permitted.

The following PR 2789 was implemented in supervisor-2021.04.3 and now checks Supervisor’s source-code for modifications. If it encounters any, it marks the instance as being unsupported and unhealthy.

Although the PR claims the following:

People can run what ever they want, it should just not show as supported.

It didn’t meet its stated goal; you can’t run whatever you want. Now that the system is deemed to be unhealthy it prevents you from installing additional Add-ons.

In addition, of the two systems I have (both running Home Assistant Supervised on Debian) the one on an RPI3 prevented me from opening the existing, installed Add-ons. I’m not sure why it behaved differently from the other (which is installed on a generic Intel-based machine) but it certainly got my attention because that’s how I discovered it was unhealthy due to ‘source_mods’.

I suspect the PR was introduced to protect users from malicious changes to Supervisor. However, anyone able to make such changes can also disable the new self-checking function (thereby rendering the PR moot).

I don’t want to disable the self-check function in order to throttle connectivity/version checks, so I’ve asked the development team to consider making the intervals user-definable. For example, the mods I made (consisting of simply altering two integer values) reduce the connectivity check to every few hours and the version check to once a day.

On a separate note, if you were the victim of malicious changes to Supervisor, there’s no ha supervisor command to replace the existing Supervisor docker image with a fresh copy (update won’t work if you’re already running the latest version, neither will repair). I imagine you’ll have to use docker pull assuming you know something about docker while you’re freaking out that you somehow managed to install a hacked copy of Supervisor.

db1964 · May 13, 2021, 9:00am

Just wanted to drop a about this , I’m as frustrated by home assistants near compulsive version checking as you are. Hopefully someone will acknowledge this issue and allow the system to chill !

123 · May 13, 2021, 12:19pm

The linked GitHub Issue (above) explains how I mitigate it now. The first post in the Issue’s thread describes the two integer values that I change in tasks.py. The last post describes the Home Assistant CLI command to disable content checking (i.e. refrain from detecting source code modifications). The result is that my instance is flagged as unsupported but not unhealthy. That arrangement allows the patched source code and permits the use of the Add-on store and to upgrade all software.

Ideally, additional control should be provided over Supervisor’s configuration, including the frequency of connectivity and version checks. If you have not voted for the following Feature Request yet, please consider it:

https://community.home-assistant.io/t/allow-some-degree-of-control-over-supervisor/306830/7

silverl · December 28, 2021, 6:56pm

I know this is old, but I stumbled on the same issue today for the version.home-assistant.io domain. Lots and lots of DNS requests in my Pi-Hole.

I thought of one new piece of info that hasn’t been discussed yet.

The frequency of a DNS lookup is related to the TTL of the DNS record. You won’t see a record in the PiHole for every call HA makes to the version.home-assistant.io domain.

Any network client making a request (like home assistant) will query DNS to look up the IP address of the remote server. The DNS response will contain a value called the “time-to-live” or TTL. The TTL is the duration that a client should cache the result. If the client makes another request before the TTL expires, the client’s OS should serve up the response from its own DNS cache.

Once the TTL expires, the next call to that server will cause a new DNS lookup.

Looking at the TTL for version.home-assistant.io: MX Toolbox DNS Lookup

It has a rather short TTL of 5 minutes. This explains why we see so many DNS requests. The cache period is very short.

Too short?

Valid reasons for a short cache period might be that the owner feels they might need to relocate the services, or fail over quickly in the event of an incident.

CyanAutomation · December 29, 2021, 7:51pm

Looking at Pi-Hole, I’m seeing a LOT of hits to version.home-assistant.io. This is 3,500 over the last 24 hours. That’s crazy and excessive.

hmadsen · December 29, 2021, 8:42pm

Found this thread because i saw an enormous amount of DNS requests from HA, to my Pihole DNS.

This request for DNS to “version.home-assistant.io” is easily the “top permitted domain” with 5 times as many requests as number 2 in the list.

I currently have 2561 requests made in the last approximately 24 hours.

I find it highly unlikely that HA is unable to function properly, if it is not able to check the version or check if it is online, THAT often.

UngluedChalice · December 30, 2021, 12:36pm

I"m in the same spot, but I have 18,000 requests to version.home-assistant.io, presumably in the last 24 hours, as I’m pretty sure that’s what my pi-hole shows by default. So that’s a request every 5 seconds.