iCloud3 v2.2.1 Device Tracker-Advanced tracking features, iOS App monitoring

gcobb321 · October 26, 2020, 11:43pm

@thoughton
Those options inform you when that type of activity takes place. They do not affect any iCloud3 tracking abilities and can be turned off. I recommended they be turned on to make sure iCloud3 is recognizing those types of triggers while you are getting up and running. If you want to still see the notifications but without the sound, go to the iPhone Settings > Notifications > Home Assistant App and turn off Sounds.

thoughton · October 26, 2020, 11:50pm

Excellent, thanks!

gcobb321 · October 28, 2020, 4:20pm

v2.2.1e Development Version - Release Candidate 2

I have just uploaded v2.2.1e to the iCloud3 GitHub repository Development Directory here. Download the .zip file, unzip it into the iCloud3 directory and restart HA.

Item #1 is the biggest change. iCloud3, when using the iOS App tracking_method, will now request location information from the iOS App just like it does with iCloud Location Services. This will provide another means of tracking your phone when there are major issues with the iCloud authentication/verification.

Change Log:

Enhancement - Updated the iOS App tracking method to request the phone’s location when the Next Update Time is reached in the same manner as the iCloud tracking methods. Using the tracking method ‘iosapp’ eliminates all iCloud account polling, relies only on the data provided by the iOS app and provides a reliable tracking method when access to the iCloud accounts is limited due to 2fa authentication/verification issues. At night, the iOS App usually shuts down when the phone is asleep. When the phone becomes inactive and is not responding to location requests, a message is displayed in the Event Log and on the iCloud3 information screen, the interval is increased until the phone wakes up and the iOS App becomes active again. When iOS App wakes up and begins responding to the location request, polling will automatically resume. Note: If the iOS App is running in the background, it still may not wake up and respond to the location requests. In that case, the iOS App must be restarted. iCloud3 will detect the ‘launch’ trigger issued by the iOS App and begin polling as normal.
Bug fix - The iOS App device_tracker entity contained the zone’s friendly name on enter/exit events while iCloud3 used the zone name for it’s device_tracker entity. This led to extra zone triggers zone mismatch errors when the names were different. This has been fixed.
Updated - Changed the way zones are handled to better support overlapping zones, particularilly when you have two zones with a center at the same location with different sizes. For example, your Home zone has a radius of 100m and you have a second zone at the Home zone’s location with a radius of 500m. The larger zone will help wake up the iOS App before you reach the Home zone. Another example is having a city size zone to display the city’s name instead of Away in the Zone and State entities on the Event Log and iCloud3 information screen.
New parameter - ‘display_zone_name: True/False’, Default: False. iCloud3 (and the iOS App) displays the zone’s friendly name in the Zone and iOSApp state fields on the Event Log and the iCloud3 information screen. If the friendly name is too long to be displayed, it will be truncated. With this parameter, the zone’s name rather than it’s friendly name, is displayed.
New parameter - ‘time_format: 12/24’, Default depends on the unit_of_measurement (mi=12, km=24). This parameter overrides the unit_of_measurement time format.
Enhancement - Added error handling to the iCloud authentication/verification process.

poudenes · October 29, 2020, 9:11am

Restart HA now after the update.
Before I restart HA also 2 times and with last one I get message to do verification… BUT the system was still working. Got all my info back and the log looks ok???

HA now is rebooted after the update. Got a popup on my iMac for verification code. in HA got a notification to select a trusted device. But hit the approve button in iMac and the notification was gone
iCloud3 is working without the trusted device selection procedure.

Sibiu · October 30, 2020, 7:37pm

Apologies if this is a silly question but I cannot find the answer and just want to check if I am even trying to solve the right problem. I am using fmf with currently just myself set up as a user. When you run it however it says no tracked devices.

In the log I can see the following:
FmF contact list email addresses found in the FindMy app for [email protected] >
• [email protected]
• [email protected]

Should I not see [email protected] on the bullet pointed list too? Is if so any idea why it is missing?

alexh1285 · October 30, 2020, 9:27pm

did you add your address in the device tracker platform?

is your Find My Phone functionality enabled on your iphone ?

Sibiu · October 30, 2020, 11:20pm

Yes, my phone address is in the configuration as a tracked device, if that is what you mean, and I can successfully log into iCloud. What I can see in the bullet points in the log are all the friends that I can see on the fmf app on my phone. I know my location is being shared as they can see me on their phones.

My first question is in your logs for that line can you see your email address twice, once at the top and once on the list part? If you can I have an Apple config issue, if not I have a home assistant config issue

gcobb321 · October 31, 2020, 12:22am

@Sibiu
You have an Apple config issue. ICloud3 receives all of the emails of the people that have accepted the invitation to be tracked and lists all it has received so you can tell if the tracked device is actually in the list when you have problems.

I would go into the FindMe app, add yourself to the people list, get an invitation and accept it just to make sure your info is in the list. Also make you are sharing your location in the Settings> Privacy>Location

Sibiu · October 31, 2020, 6:36am

Thanks, but can’t get that to work, when I add my address in the FindMe app it gives me an error saying you cannot share with yourself. Location sharing is enabled in settings.

gcobb321 · October 31, 2020, 7:40am

@Sibiu
I have the following setup on my phone:

Settings > Privacy > Location Services
— Location Services is On
Settings > Privacy > Location Services > Share My Location
— My Location is This Device
— Share My Location is On
Settings > Privacy > Location Services > Share My Location > Find My Phone
— Find My Phone is On
— Find My network is On
— Under Family, I see my wife
— Under Friends, I see myself and my second iCloud non-2f account user name

I see myself, my wife and my non-2fa name in FindMy > People and FindMy > Me > Share My Location is On.

On the iCloud3 Event Log, for my 2fa account I am using, I see my 2fa email, my wife’s email and my second non-2fa account email.

To see the raw data returned from iCloud during startup that is decoded to extract the email addresses, put log_level: debug+rawdata in the configuration parameter for iCloud3. It will be in the HA log file. It’sa mess to wade through but, using a text editor, you can search for the email addresses and see if they are all there. They will be in the ‘Friends’ entries.

poudenes · October 31, 2020, 4:09pm

Hi @gcobb321,

This is the 3rd time I see this error. The first time it let crash MariaDB at night and saw it in the morning.
Don’t know what this is.

2020-10-31 16:11:11 ERROR (Recorder) [homeassistant.components.recorder] Error executing query: (MySQLdb._exceptions.DataError) (1406, "Data too long for column 'attributes' at row 1")
[SQL: INSERT INTO states (domain, entity_id, state, attributes, event_id, last_changed, last_updated, created, old_state_id) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)]
[parameters: ('sensor', 'sensor.icloud3_event_log', 'peter_iphone:Sat, 10/31, 16:01:10', '{"log_level_debug": "", "filtername": "Peter", "update_time": "Sat, 10/31, 16:01:10", "names": {"peter_iphone": "Peter", "cenkay_xr": "Cenkay_Xr"}, " ... (111395 characters truncated) ... ', \'\', \'\', \'\', \'\', \'^^^ Initializing iCloud3 v2.2.1e > Saturday, Oct 31\'], [\'00:00:00\', \'\', \'\', \'\', \'\', \'\', \'Last Record\']]"}', 13414815, datetime.datetime(2020, 10, 31, 15, 1, 10, 494582, tzinfo=datetime.timezone.utc), datetime.datetime(2020, 10, 31, 15, 1, 10, 494582, tzinfo=datetime.timezone.utc), datetime.datetime(2020, 10, 31, 15, 1, 10, 494716, tzinfo=datetime.timezone.utc), 12914612)]
(Background on this error at: http://sqlalche.me/e/13/9h9h)
2020-10-31 16:11:11 ERROR (Recorder) [homeassistant.components.recorder] Error saving events: (MySQLdb._exceptions.DataError) (1406, "Data too long for column 'attributes' at row 1")
[SQL: INSERT INTO states (domain, entity_id, state, attributes, event_id, last_changed, last_updated, created, old_state_id) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)]
[parameters: ('sensor', 'sensor.icloud3_event_log', 'peter_iphone:Sat, 10/31, 16:01:10', '{"log_level_debug": "", "filtername": "Peter", "update_time": "Sat, 10/31, 16:01:10", "names": {"peter_iphone": "Peter", "cenkay_xr": "Cenkay_Xr"}, " ... (111395 characters truncated) ... ', \'\', \'\', \'\', \'\', \'^^^ Initializing iCloud3 v2.2.1e > Saturday, Oct 31\'], [\'00:00:00\', \'\', \'\', \'\', \'\', \'\', \'Last Record\']]"}', 13414815, datetime.datetime(2020, 10, 31, 15, 1, 10, 494582, tzinfo=datetime.timezone.utc), datetime.datetime(2020, 10, 31, 15, 1, 10, 494582, tzinfo=datetime.timezone.utc), datetime.datetime(2020, 10, 31, 15, 1, 10, 494716, tzinfo=datetime.timezone.utc), 12914612)]
(Background on this error at: http://sqlalche.me/e/13/9h9h)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.DataError: (1406, "Data too long for column 'attributes' at row 1")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/recorder/__init__.py", line 454, in _commit_event_session_or_retry
    self._commit_event_session()
  File "/usr/src/homeassistant/homeassistant/components/recorder/__init__.py", line 509, in _commit_event_session
    self.event_session.flush()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2536, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2678, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2638, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 419, in execute
    n.execute_aggregate(self, set_)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 697, in execute_aggregate
    persistence.save_obj(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 239, in save_obj
    _emit_insert_statements(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1135, in _emit_insert_statements
    result = cached_connections[connection].execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.DataError: (MySQLdb._exceptions.DataError) (1406, "Data too long for column 'attributes' at row 1")
[SQL: INSERT INTO states (domain, entity_id, state, attributes, event_id, last_changed, last_updated, created, old_state_id) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)]
[parameters: ('sensor', 'sensor.icloud3_event_log', 'peter_iphone:Sat, 10/31, 16:01:10', '{"log_level_debug": "", "filtername": "Peter", "update_time": "Sat, 10/31, 16:01:10", "names": {"peter_iphone": "Peter", "cenkay_xr": "Cenkay_Xr"}, " ... (111395 characters truncated) ... ', \'\', \'\', \'\', \'\', \'^^^ Initializing iCloud3 v2.2.1e > Saturday, Oct 31\'], [\'00:00:00\', \'\', \'\', \'\', \'\', \'\', \'Last Record\']]"}', 13414815, datetime.datetime(2020, 10, 31, 15, 1, 10, 494582, tzinfo=datetime.timezone.utc), datetime.datetime(2020, 10, 31, 15, 1, 10, 494582, tzinfo=datetime.timezone.utc), datetime.datetime(2020, 10, 31, 15, 1, 10, 494716, tzinfo=datetime.timezone.utc), 12914612)]
(Background on this error at: http://sqlalche.me/e/13/9h9h)

alexh1285 · October 31, 2020, 4:54pm

@Sibiu

I have the exact same settings on my iphone too, and everything works fine.

Do you have the same?
Are you on ios 12/13/14?

Sibiu · October 31, 2020, 10:23pm

@alexh1285
I am running ios 13.7 and have exactly the same settings on my phone as you guys. Doing an update to ios 14.1 now just in case this is an issue, but it doesn’t seem to be the case.

@gcobb321
The settings are as you suggested. I tried adding the log_level to configuration.yaml as you suggested and it dod’t generate any extra information, so that is likely my error and I will try again tomorrow when I have more time

gcobb321 · October 31, 2020, 10:50pm

@poudenes
Put the following in your configuration.yaml file to exclude sensor.icloud3_event_log from being added to the HA history dictionary. Go here for more information about the recorder parameter.

recorder:
  exclude:
    entities:
      - sensor.icloud3_event_log

poudenes · November 1, 2020, 3:47pm

I got lots of verification request in Watch and iPhone. I removed the file ./storage/icloud/ restart HA.
It create a new file, but I don’t get the verification procedure.

the file contains only this:

root@hassio:/usr/share/hassio/homeassistant/.storage/icloud# cat petexxxxxxxxxxcom
#LWP-Cookies-2.0
Set-Cookie3: X-APPLE-WEBAUTH-USER="\"v=1:s=1:d=1092288269\""; path="/"; domain=".icloud.com"; path_spec; domain_dot; secure; expires="2020-11-15 15:40:55Z"; HttpOnly=None; version=0
Set-Cookie3: X_APPLE_WEB_KB-COZU0VSAWHYI8Q8EQLGC1DTNMCC="\"v=1:t=AQ==BST_IAAAAAAABLwIAAAAAF-e1wcRDmdzLmljbG91ZC5hdXRovQC9Q_pPHazHAQMx-0dkiHDwzfWlC9Zc4tq1MnqbHzRDJCRyZwk18_7axxxxxxxxxxxxxxxxx83BY9mKyiuM4nPZyNaZDJv-BXdqdYxxxxxxxxxxxxOtbv99rmJXJFjIxzhJF32qza2ZMoPg~~\""; path="/"; domain=".icloud.com"; path_spec; domain_dot; secure;
expires="2020-12-31 15:40:55Z"; HttpOnly=None; version=0

Also I don’t see a ERROR that I must do verification procedure…

jdmartinez · November 1, 2020, 7:34pm

I think it’s an Apple error. Find app doesn’t find any device. Maybe Apple servers are down

gcobb321 · November 2, 2020, 4:25pm

iCloud3 v2.2.1 has been released

iCloud3 v2.2.1 includes enhancements, bug fixes and some new configuration parameters. Details are in the iCloud3 User Manual here but the major ones are:

Updates to the iOS App tracking method that support phone polling requests and updates like the Family Sharing and Find-my-Friends tracking methods.
Improved zone/zone name display and handling.
New configuration parameters for displaying zone names in the Event Log and using a 24-hour time format.
Improved handling of old locations.
Ability to display the iCloud3 User Manual from the Event Log.
Improved support for how the iOS App handles zone change notifications.
Bug fixes.

gcobb321 · November 2, 2020, 4:39pm

Information about the iCloud account authentication and verification issues.

This is a known problem within HA and an alert has been released (thanks @bobvmierlo for spotting it).

iCloud causes Apple ID sign-in notifications every 30 min
Since Oct 9, 2020 the iCloud integration prompts a sign-in notification every 30 min. It is not compulsory to respond to the notification, the integration is still working during this time.>
A fix will be proposed to avoid this issue but we don’t know when yet.

Link here
There are people working on updating the pyicloud.py interface module between the iCloud integration and iCloud Location Service to support native authentication/verification using the 6-digit code from the notification popup window Apple issues when a new device logs into the iCloud account. It is going through testing and so far, Apple has not asked for a reverification after the verification code was entered 4-days ago. I have been in touch with the developers and we will incorporate it into the pyicloud_ic3.py customized version used by iCloud3 for additional testing when it is ready. I’ll let you know when that has been done and it is available for testing.

Gary

bastero · November 4, 2020, 4:48pm

@gcobb321 Hi Gary, I believe I’ve found an issue with the latest version i.e. V2.2.1 regarding the capitalization of the first character of the device_tracker State name(s) being assigned, this is affecting an automation that is monitoring a defined device_tracker group.

I have two device trackers (device_tracker.iphoneexpress & device_tracker.garbineiphone) that when grouped ( group.awesome_people) never change the State of the group even with both devices parked in their home zone. The group’s State never changes to ‘home’ from ‘not_home’. This was not the case with previous icloud3 releases.

I’m not sure if this is related but for device_tracker.garbineiphone, when stationary within her work zone, the State is ‘WorkGarbine’, whereas for device_tracker.garbineiphone_app the State is 'Work Garbine’ (includes a space).

All device trackers being monitored have ‘State’ that start with a capitalized letter. This is what I believe is not being evaluated properly within the device_tracker group??

Thanks for a great custom component!

gcobb321 · November 4, 2020, 4:50pm

iCloud3 has been updated to v2.2.1a (11/4/2020)

The v2.2.1 released was updated rather than issuing a new release since this only effected those who were using the device_tracker_[DEVICENAME] state field in group settings.

If you have not updated from v2.2.0, this will be installed with the normal HACS update.
If you have already updated to v2.2.1, open the HACS integration, search for iCloud3 and select reinstall. Or download the icloud3.zip file in the iCloud3 repository and unzip it to the iCloud3 directory.

Change Log

Fixed a bug where the device_tracker/[DEVICENAME] state was being set to Home instead of home or Away instead of not_home.
Depreciated the zone_name1, zone_name2 and zone_name3 entities. They have been renamed to zone_name, zone_title and zone_fname respectively. See the 2.4 Using Sensors in the iCloud3 User Manual for more information. You should change any automations and scripts that use the zone_name# entities.