Composite Device Tracker Platform

pnbruckner · November 5, 2022, 7:57pm

Actually, I don’t know about the behavior you described. I’ve read the code for the person integration, so I know basically how it uses its “input trackers”, although I will admit I haven’t reviewed it very recently.

I think the main difference is that the composite’s approach is that “new data” is better than “old data”, rather than trying to prioritize the types of the various input data. It also recognizes that for most non-GPS based inputs (routers, pings, blue-tooth, binary sensors, …) they are much better at determining when a device comes home than when it leaves, and by default, only uses the home states of those types of devices. I did add a new feature recently, though, that allows one to configure the composite to use all states of any given input if that makes more sense for the user’s scenario.

Another thing the composite does that I’m pretty sure the person integration doesn’t is to use attributes of input devices that indicates how “old” its GPS data is. E.g., both the life360 and google_maps integrations (which I happen to use) have last_seen attributes. So, if one of these entities updates for some other reason, but the last_seen “timestamp” hasn’t changed, it won’t be looked at as necessarily newer.

To describe that another way, let’s say input 1 updates, so the composite takes its data and updates itself. Now input 2 updates and its last_seen is newer than the last_seen from input 1, meaning it is newer data. In this case the composite takes the data from input 2 and updates itself. Now the input 1 entity updates for some other reason, but its last_seen hasn’t, and is therefore still older than the recent last_seen from input 2. In this case the composite ignores that update and retains the last information from input 2, because it’s newer and is a better representation of where the device actually is.

Agreed. Like I said, I use Life360 & Google Maps, and using both on the same phone results in more up-to-date data, and for the reasons given above (mainly checking the last_seen attribute), the composite does a better job of always using the most up-to-date data than the person integration does (again, at least as far as I know since the last time I reviewed its code.)

Agreed. And if I cared about that feature I’d use the person integration “on top of” the composite integration to get the best of both worlds.

Edwin_D · November 5, 2022, 7:57pm

True. Thanks for your take on the differences.

Edwin_D · November 5, 2022, 8:05pm

That does sound like an improvement indeed. Thank you for taking the time to explain.

As for binary wifi/router/ping device trackers: those are often very good and fast at registering coming home, but often very bad at detecting away as devices tend to turn off wifi to save battery. Also some routers are slow at assuming offline. How do you prioritize binary home/not home? The person integration seems to get that bit right.

The tendency to add devices to person will be worse when using your integration though, because it may add phone and ipad again, which would be unwanted using your integration.

pnbruckner · November 5, 2022, 8:24pm

Sorry to be so verbose, but just to drive the point home, in the scenario I described above the person integration would use the second change from the input 1 device, even though it’s older, mainly because it’s not looking at the last_seen attribute. So, if the data points were A (input 1 first update) and B (input 2 update), whereas the composite would show A → B, the person would show A → B → A.

It doesn’t. It simply only uses state changes from these devices when the new state is “Home” (i.e., home).

Again, the person integration (which I just rechecked) updates whenever any of the inputs change, and then tries to prioritize based on the current states of all the inputs at that time. And it only looks at the last_updated state field, which is not necessarily a valid representation of how new the data is. (FYI, the composite tracker will fall back to this same data if the input entity that changed doesn’t have a last_seen attribute.)

The composite, on the other hand, only cares about the new state of the input that just changed, and its own state, including its recording of how old its current data is. If the new state has data that is newer, it’s used (except, again, for non-GPS devices, that only use new states that are home, unless configured to use all states of the given input.)

One last point. The person integration does have logic to ignore unavailable and unknown states from its input devices. However, that can cause it to change from newer data to older data because of the way it works. Also, if it has only one input device, it would become unknown. The composite handles both of these scenarios better because it simply retains the last known good data. Basically, it always tries to use the most up-to-date data it has seen, and the lack of data is not new data. And it also has a last_seen attribute, so you can always tell just how old its data is.

Ildar_Gabdullin · November 5, 2022, 8:37pm

@pnbruckner
With beta platform none of my trackers is valid.
Here are 3 of them (12 in total):

composite:
  tz_finder: timezonefinder<6
  tz_finder_class: TimezoneFinderL

  trackers:
    - name: iphone_5s_composite
      time_as: local 
      entity_id:
        - device_tracker.iphone_5s_life360
        - device_tracker.iphone_5s_asuswrt
        - device_tracker.iphone_5s_iphonedetect
        - device_tracker.iphone_5s_companion
        - device_tracker.iphone_5s_traccar

Now a strange thing with a logger.
With my “default: warn” setting I only get this in the Log (for each tracker):

2022-11-05 22:58:02.001 ERROR (MainThread) [homeassistant.components.device_tracker] Error adding entities for domain device_tracker with platform composite
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 428, in async_add_entities
await asyncio.gather(*tasks)
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 692, in _async_add_entity
await entity.add_to_platform_finish()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 788, in add_to_platform_finish
await self.async_added_to_hass()
File "/config/custom_components/composite/device_tracker.py", line 271, in async_added_to_hass
await self.hass.async_add_executor_job(setup_scanner)
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/config/custom_components/composite/device_tracker.py", line 269, in setup_scanner
self._scanner = CompositeScanner(self.hass, self._scanner_config, self._see)
File "/config/custom_components/composite/device_tracker.py", line 312, in __init__
entities = config[CONF_ENTITY_ID]
KeyError: 'entity_id'

With this settings:

  default: info
  logs:
    custom_components.composite: debug
    homeassistant.core: debug

I only get “core” records.

pnbruckner · November 5, 2022, 9:45pm

Any chance you could share one of the corresponding entries from .storage/core.config_entries?

pnbruckner · November 5, 2022, 9:55pm

FYI, I’ve only tested with HA 2021.3.3 / Python 3.8 and HA 2022.11.0 / Python 3.9. I see you’re using Python 3.10. What version of HA are you using?

Ildar_Gabdullin · November 5, 2022, 10:23pm

2022.11.1
Could it be a reason of these errors?

Do you still need this?

pnbruckner · November 5, 2022, 10:53pm

I don’t know why it’s failing like that so I’m trying to get details. Yes, that would help.

Ildar_Gabdullin · November 5, 2022, 11:04pm

The “core.config_entries” only contains entries for 3 items (out of 12) - and these items related to entities which MAY BE unavailable:
– 2 of them belong to mobile phones with Life360 - which does not work in Russia (corresponding device_trackers = “unavailable”);
– 1 - belong to iPad mini (which does not work for 1 year).

      {
        "entry_id": "f30bceffc4c51fca0019a8230eaffa30",
        "version": 1,
        "domain": "composite",
        "title": "Phone Boris Composite (from configuration)",
        "data": {
          "name": "Phone Boris Composite",
          "id": "phone_boris_composite"
        },
        "options": {
          "time_as": "local",
          "entity_id": [
            {
              "entity": "device_tracker.phone_boris_life360",
              "all_states": false
            }
          ],
          "require_movement": false
        },
        "pref_disable_new_entities": false,
        "pref_disable_polling": false,
        "source": "import",
        "unique_id": "phone_boris_composite",
        "disabled_by": null
      },
      {
        "entry_id": "b56cbb9497451a8c07f1f349afb18586",
        "version": 1,
        "domain": "composite",
        "title": "Phone Galina Composite (from configuration)",
        "data": {
          "name": "Phone Galina Composite",
          "id": "phone_galina_composite"
        },
        "options": {
          "time_as": "local",
          "entity_id": [
            {
              "entity": "device_tracker.phone_galina_life360",
              "all_states": false
            }
          ],
          "require_movement": false
        },
        "pref_disable_new_entities": false,
        "pref_disable_polling": false,
        "source": "import",
        "unique_id": "phone_galina_composite",
        "disabled_by": null
      },
      {
        "entry_id": "8f05856a582fbbed16cb2782906d7746",
        "version": 1,
        "domain": "composite",
        "title": "Ipad Mini 2 Composite (from configuration)",
        "data": {
          "name": "Ipad Mini 2 Composite",
          "id": "ipad_mini_2_composite"
        },
        "options": {
          "time_as": "local",
          "entity_id": [
            {
              "entity": "device_tracker.ipad_mini_2_life360",
              "all_states": false
            }
          ],
          "require_movement": false
        },
        "pref_disable_new_entities": false,
        "pref_disable_polling": false,
        "source": "import",
        "unique_id": "ipad_mini_2_composite",
        "disabled_by": null
      }

I do not understand why only these entries here.

I noticed that the “core.entity_registry” file also contains entries for same 3 entities.
Could these entries be kept from a previous run of HA - with the old platform?
That time these 3 entities COULD BE unavailable (due to reasons I explained above), other 9 entities were good.

pnbruckner · November 5, 2022, 11:19pm

Unfortunately, I’ll be kind of unavailable myself this weekend. I probably won’t be able to really work on this until Monday. I may have to add more debug statements because so far based on the data you’ve shared it doesn’t make sense.

Did you get any messages or persistent notifications about conflicting IDs?

Ildar_Gabdullin · November 5, 2022, 11:30pm

Yes!

And I do not know how to remove them - unless I am supposed to remove them from “known_devices.yaml”.

Also, this “composite” entities are strangely listed - like 3 of them (same as supposed to be “unavailable”) may be edited via UI.

Ildar_Gabdullin · November 5, 2022, 11:32pm

No problem! I will return to the old platform.
This is just about renaming “composite_old.dis” → “composite_old.yaml”, “composite_new.yaml” → “composite_new.dis” - which is possible since you maintain a YAML config!

finity · November 5, 2022, 11:37pm

I just tried it on the beta and my device tracker is unavailable with this error in the logs:

2022-11-05 18:10:44.307 ERROR (MainThread) [homeassistant.components.device_tracker] Error adding entities for domain device_tracker with platform composite
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 428, in async_add_entities
    await asyncio.gather(*tasks)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 691, in _async_add_entity
    await entity.add_to_platform_finish()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 776, in add_to_platform_finish
    await self.async_added_to_hass()
  File "/config/custom_components/composite/device_tracker.py", line 271, in async_added_to_hass
    await self.hass.async_add_executor_job(setup_scanner)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/composite/device_tracker.py", line 269, in setup_scanner
    self._scanner = CompositeScanner(self.hass, self._scanner_config, self._see)
  File "/config/custom_components/composite/device_tracker.py", line 312, in __init__
    entities = config[CONF_ENTITY_ID]
KeyError: 'entity_id'
2022-11-05 18:10:44.314 ERROR (MainThread) [homeassistant.components.device_tracker] Error while setting up composite platform for device_tracker
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 289, in _async_setup_platform
    await asyncio.gather(*pending)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 428, in async_add_entities
    await asyncio.gather(*tasks)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 691, in _async_add_entity
    await entity.add_to_platform_finish()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 776, in add_to_platform_finish
    await self.async_added_to_hass()
  File "/config/custom_components/composite/device_tracker.py", line 271, in async_added_to_hass
    await self.hass.async_add_executor_job(setup_scanner)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/composite/device_tracker.py", line 269, in setup_scanner
    self._scanner = CompositeScanner(self.hass, self._scanner_config, self._see)
  File "/config/custom_components/composite/device_tracker.py", line 312, in __init__
    entities = config[CONF_ENTITY_ID]
KeyError: 'entity_id'

Ildar_Gabdullin · November 5, 2022, 11:42pm

At least the 1st error seems to be same as mine.

pnbruckner · November 5, 2022, 11:50pm

Ok. I see you’re using Python 3.10 as well. I’ll have to give that a try, although I can’t imagine how that could have anything to do with it. Hmm.

pnbruckner · November 5, 2022, 11:53pm

Exactly.

Hopefully you don’t run into trouble going back. If so, try removing the entries from .storage, but that shouldn’t be necessary, I think.

Ildar_Gabdullin · November 5, 2022, 11:55pm

@pnbruckner
VICTORY

Removed “composite” entities from “known_devices”.
Restarted HA.
Removed restored entities from Settings → Entities.
Restarted HA.
Now at least 9 entities are VALID !!! (3 = unknown - as expected)

Testing next…
Thanks a lot!

pnbruckner · November 6, 2022, 12:23am

Well I’m glad it’s working. You should only have had to do the first two steps. And that exception should not have happened either. I’ll see if I can figure out why it happened.

pnbruckner · November 6, 2022, 2:30am

Did you also get any messages or persistent notifications about conflicting IDs?