How to remove a call_later in a unit test

Development
1

Morning,

I am using a call_later in a custom component in an entity, however I cannot seem to clean this up when the component is removed. It complains about me calling the cleanup in the event loop. If I don’t clean it up then I get this error in the unit tests:
Failed: Lingering timer after job <Job call_later 2 HassJobType.Executor

I tried to override all the remove calls I could see in the entity to cleanup the timeouts, but they all fail on the event loop issue. Is there a way you should cleanup timeouts as the system shuts down or the entity is destroyed?

Thanks,
David.

2

Post a link to your code so I can have a better look at what you are doing.

Fundamentally, i would use

async def async_added_to_hass(self):
        """Run when this Entity has been added to HA."""
    self._task = asyncio.call_later(MY_FUNC)

to assign your async task to a variable

    async def async_will_remove_from_hass(self):
        """Entity being removed from hass."""
        await self._task.cancel()

To cancel it on shutdown.

However, it might be something else in your code not unloading the integration properly.

3

I am using call_later from homeassistant.helpers.events

If I do this and run it with the cancel call it gives an error of:
raise RuntimeError(“Cannot be called from within the event loop”) from async_py line 56

If I try and use async callbacks I get errors about the event loop itself. Not entirely sure how to post to the event loop? If I use something that doesn’t wrap the event loop, trying to unwind and use run_callback_threadsafe also gives an error about trying to run on the wrong event loop.

4

Ok, so the events async_call_later function returns the cancel so you would just call self._task(). In which case, you should be doing this.

async def async_added_to_hass(self):
    """When entity is added to hass."""
    self.async_on_remove(async_call_later(hass, DELAY, MY_FUNC))

It maybe however, that there is a better way to do this. It’s much easier for me to help if I can see your code.

5

If I do that it will error with raise RuntimeError(“Cannot be called from within the event loop”) from async_py line 56

I am writing this:

Looking at the clear and extended timeouts in the select code. From searching in the home assistant code itself, no one seems to cleanup their timeouts on remove, but if you don’t the test definately errors.

6

If I use async_call_later I get this error:
RuntimeError: Non-thread-safe operation invoked on an event loop other than the current one

It gives the error when I call async_call_later, but if I use the normal call later then it is impossible to cancel the timer.

    def _async_update_with_job(self):
        self.hass.async_add_job(self._update_state)

    def _set_clear_timeout(self) -> None:
        if self._clear_timeout_callback:
            self._remove_clear_timeout()

        timeout = self._get_clear_timeout()

        self.logger.debug("%s: Scheduling clear in %s seconds", self.area.name, timeout)
        self._clear_timeout_callback = async_call_later(
            self.hass,
            timeout,
            self._async_update_with_job,
        )
7

Ok, let me have a look at this tomorrow. From a quick scan I can see a few areas to dig into more.

8

As said, will have a look through your code and make some suggestions but i think you have a misunderstanding of async.

If you are using an async function such as async_call_later, it must be calling an async function or a function decorated with @callback to run in the hass event loop.

I think you are also making your code very complicated to follow by having too many function calls on the way. Ie using async_call_later to add a task to the loop calling a function to add another task to the loop (async_add_job) to then run your actual function of self._update_state.

If _update_state is either an async function or decorated with @callback, you can call it directly from your async_call_later. Ie

    @callback
    def _update_state(self) -> None:
        # do function stuff
    
    def _set_clear_timeout(self) -> None:
        if self._clear_timeout_callback:
            self._remove_clear_timeout()

        timeout = self._get_clear_timeout()

        self.logger.debug("%s: Scheduling clear in %s seconds", self.area.name, timeout)
        self._clear_timeout_callback = async_call_later(
            self.hass,
            timeout,
            self._update_state,
        )
9

Yes, was adding in extra layers to see if I could get the timer to still run but not fail on the remove. Couldn’t solve it though ;(. The main issue is I cannot remove it at the end/cancel it, in the entity_remove flow. It errors about being in the wrong thread. I think it is because the timer callback is also run_thread protected, but this means it will fail in the remove flow. You can do this really easily by just adding the timer callback into the entity remove and then removing the entity.

10

Yes, had a look last night and there is quite a bit i would change to simplify this down. I think you are maybe getting lost in all your listener callbacks which end up then on threads and not in the event loop. You then have a lot running in a thread which you dont think is and this is where your problem stems from.

Don’t want to tread on any toes, but if you want to have a look at what I have done to change (not yet gone through it all), its on my fork of your repo.

Main changes I have made for you to look at.

As i traced through your startup, i could see that you were fundamentally waiting for HA to start before analysing what entities exist and making your own entities to control these like a group. However, you had a listener (which never cancelled) on each entity type for this. I have stripped this back to have a single listener in your init.py file which waits for HA to be running to then set everything up. This removes all the other listeners in your entity setups and removes some of the code in your base/magic.py that was managing this and causing threading.

I also looked at your gathering of existing entities and tried to simplify down as was very hard to follow (this was more for me to be able to see what is happening better - so use or discard as you feel).

I would note that this sets up as your version but with nothing on a thread. However, i see errors when entities update, which also exists in yours. I am next to work my way through all your entity classes (again, very complicated and hard to follow - may just be me! :grin:), where i think some of your other threading issues lie and the source of the errors when entities update and when you are trying to remove your timer.

11

This is very helpful thanks! I forked this from a different repro and already did some of the cleanups you suggested. I pulled everything out of primities and enties except one thing and moved bits down. I had issues working out what was setup where too :). The entity fixes in the magic_area and init are very helpful.