Local Deployment for SureFlap / SurePetCare Connect using only local MQTT Broker

peterl · April 17, 2021, 12:01am

I think I have made it as simple as it can be.
If you follow the instructions at

The main points are you need to setup DNS, and have MQTT discovery setup in Home Assistant and then install docker with docker-compose. Then once you setup the ini file, create the database it should just automatically start populating the sensors and switches into Home Assistant like magic.

Then you can reconfigure / move around the entities as you see fit.

Also made quite a few changes this morning to improve the cat flap code, and refactoring to use Box rather than regular dict, but re-ran it with a few weeks of logs and seems to work fine.

I would also suggest re running the database create script again or checking out how I have changed the structure for the better.

peterl · April 19, 2021, 10:03am

Added the battery for the feeder, the developers of the hardware must have been completely different companies as the messages from Pet Door and the Feeder are just completely different even the battery values… ugh.

flyize · April 19, 2021, 8:04pm

I’m trying to set this up, but keep in mind that I’m essentially 5 years old when it comes to this stuff sometimes…

Added my credentials and tried running ‘python3 mkpetlocaldb.py’ and got the following output:

no such table: devices
no such table: hubs
no such table: doors
no such table: feeders
no such table: tagmap
no such table: pets
no such table: petstate
no such table: devicecounter
Traceback (most recent call last):
  File "mkpetlocaldb.py", line 221, in <module>
    asyncio.run(petlocaldb())
  File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "mkpetlocaldb.py", line 91, in petlocaldb
    device_id = pets['status']['activity']['device_id']
KeyError: 'device_id'

kimagure · April 20, 2021, 9:18am

The pets['status']['activity']['device_id'] value is the latest activity for a pet for a specific device (feeder/pet door/cat flap), and it seems it’s missing in the reply from the surepet API.

The debuging is enabled by default in the mkpetlocaldb.py script, but please make sure the PrintDebug = True is not commented out at the beginning of the script.
Run the python script again and you should see a block of JSON data before the “no such table: devices” message. If you do not see anything, please check your credentials, because this means the cloud API call is failing.

Each item in the “pet” JSON table should have a “status/activity” item as below:

'status': {
	'activity': {
		'tag_id': XXXX,
		'device_id': YYYY,
		'where': 1,
		'since': '2021-04-07T12:24:28+00:00'
	}
}

And it seems that, at least for one pet, this information is missing. Can you check what you have?

If you want me to check, you can PM me the JSON data (please do not post it publicly, there are a few sensitive items inside)

K.

flyize · April 20, 2021, 11:38am

Yes, there was a good chunk of JSON above the text I C&Pd. I’ll parse it out and see what’s missing. Thanks!

flyize · April 20, 2021, 12:54pm

Interesting. The two cats that were put in first do not have a ‘device_id’. They’re indoor only, so their ‘since’ is months ago. Maybe they recently added the device_id?

peterl · April 20, 2021, 1:33pm

Device_id has been there for a long time, as I have seen in in my starts for ages.
The problem was I had all the logic around matching the pet assigned to the device rather than the other way around, so if you could email me the full json file I can take a look. pushed a minor update to just write empty values into the db… which you can fix up yourself using sqlite.

CREATE TABLE petstate(tag TEXT, mac_address TEXT, timestamp TEXT, state BLOB );

The only important matching is tag number to mac address of the device.
Another quick fix would probably be unassigned the animal from the device and re-assign it.

flyize · April 20, 2021, 2:02pm

Okay, so now I’m getting this error. I assumed it was a python package error, so I installed datetime, but that didn’t seem to fix it.

Traceback (most recent call last):
  File "mkpetlocaldb.py", line 233, in <module>
    asyncio.run(petlocaldb())
  File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "mkpetlocaldb.py", line 48, in petlocaldb
    sttime = datetime.now().strftime('%Y%m%d_%H%M%S')
NameError: name 'datetime' is not defined

kimagure · April 20, 2021, 3:03pm

Add this at the end of the import section at the top of the script:

from datetime import datetime

It should work.

With @peterl’s last update, the JSON data from the surepet API will be saved in a local file for easy review.

flyize · April 20, 2021, 4:36pm

Of course! It’s been a long time since I’ve messed with python at all. Anyway, now I’m getting this error:

Traceback (most recent call last):
  File "mkpetlocaldb.py", line 234, in <module>
    asyncio.run(petlocaldb())
  File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "mkpetlocaldb.py", line 231, in petlocaldb
    await sp.close_session()
AttributeError: 'SureAPIClient' object has no attribute 'close_session'

kimagure · April 20, 2021, 7:06pm

Bad luck, it seems there was a release of version 0.6.0 of surepy a few days ago, and is has some breaking changes… You probably have this version installed.

As this error occurs on the the last action of the script (to properly close the connection to the cloud API), maybe this is not a big issue and your local db has been created properly.
You can use sqlite3 to check the content of the tables (devices, hubs, doors, pets). If the content if there, you are good to go.

If the data is missing in the localdb and you are not using surepy for anything else on your server, maybe you can downgrade to the previous version 0.5.0 of surepy. The script works fine with this version.

Making mkpetlocaldb with version 0.6.0 will require some refactoring…

K.

peterl · April 20, 2021, 9:07pm

Actually I just messed up mkpetlocaldb… Now fixed it and support a command line option to import the start.json file I write during the cloud call. So you shouldn’t need to keep on hitting the cloud.

peterl · April 22, 2021, 2:14pm

Lots of updates and again updates to the db to support hub state and uptime, curfew modes, reduce the number of sensors created and add them on as extra json values to the Hub/Devices.
So you will need to re-run the script but the local json “start” file should still work just re-run it.
Also started playing with mini-graph-card and very happy with the pretty graphs it creates.

flyize · April 22, 2021, 3:53pm

Whats up with the battery readings?

peterl · April 22, 2021, 7:07pm

That’s just me running a number of days worth of logs into the stack simulating what the hub would be doing to make sure my code hasn’t broken anything.
The battery is now correctly being calculated for both the feeder and pet door. If someone emailed me msgs logs from a few days from a cat flap I could check that too.

tinuva · April 23, 2021, 1:26pm

What am I doing wrong here?
Was still on a much earlier commit, so updated today. Made a new db file, that went without issue. But pethub is not happy.

pethub             | HAMQTTIP connecting to 192.168.241.12
pethub             | Starting Pet Hub
pethub             | Connecting to Home Assistant MQTT endpoint at 192.168.241.12 port 1883
pethub             | Load Devices from pethublocal.db and create Home Assistant MQTT discovery config topics
pethub             | Loading Hub: Home hub
pethub             | Traceback (most recent call last):
pethub             |   File "pethubmqtt.py", line 384, in <module>
pethub             |     states.hub=Box({"State":Online(device.state).name, "Uptime":"0 Mins", "Name": dev, "Serial": device.serial_number,"MAC Address": mac, "LED Mode":Enabled(device.led_mode).name, "Pairing Mode":Enabled(device.pairing_mode).name })
pethub             |   File "/usr/local/lib/python3.8/enum.py", line 339, in __call__
pethub             |     return cls.__new__(cls, value)
pethub             |   File "/usr/local/lib/python3.8/enum.py", line 662, in __new__
pethub             |     raise ve_exc
pethub             | ValueError: 4 is not a valid Enabled
pethub exited with code 1

sqlite> select * from devices;
0000801F121D6719|1|Home hub|H004-0088731||||{"device": {"hardware": 3, "firmware": 2.43}}
D98122FEFFEC1054|3|Kitchen||2.9012|-75|-76.4|{"lcd": {"hardware": 1, "firmware": 1}, "rf": {"hardware": 4, "firmware": 0.16}}
sqlite> select * from hubs;
0000801F121D6719|4|0|1|0
sqlite> select * from doors;
D98122FEFFEC1054|0|12:01|12:02|0|000000
sqlite> select * from pets;
711033360D00|Ziva|2
4C4B0A355500|Jessie|2
992.003000057158|Vuur|1
992.003000057159|Vlam|1

Ran it again with debug:

pethub             | Starting Pet Hub
pethub             | Connecting to Home Assistant MQTT endpoint at 192.168.241.12 port 1883
pethub             | SQLite init database response: {'devices': [{'name': 'Home hub', 'product_id': 1, 'mac_address': '0000801F121D6719', 'serial_number': 'H004-0088731', 'version': '{"device": {"hardware": 3, "firmware": 2.43}}', 'state': 1, 'battery': '', 'led_mode': 4, 'pairing_mode': 0, 'curfewenabled': None, 'lock_time': None, 'unlock_time': None, 'lockingmode': None, 'bowl1': None, 'bowl2': None, 'bowltarget1': None, 'bowltarget2': None, 'bowltype': None, 'close_delay': None}, {'name': 'Kitchen', 'product_id': 3, 'mac_address': 'D98122FEFFEC1054', 'serial_number': '', 'version': '{"lcd": {"hardware": 1, "firmware": 1}, "rf": {"hardware": 4, "firmware": 0.16}}', 'state': None, 'battery': '2.9012', 'led_mode': None, 'pairing_mode': None, 'curfewenabled': 0, 'lock_time': '12:01', 'unlock_time': '12:02', 'lockingmode': 0, 'bowl1': None, 'bowl2': None, 'bowltarget1': None, 'bowltarget2': None, 'bowltype': None, 'close_delay': None}], 'pets': [{'name': 'Ziva', 'species': 2, 'device': 'Kitchen', 'product_id': 3, 'state': 2}, {'name': 'Jessie', 'species': 2, 'device': 'Kitchen', 'product_id': 3, 'state': 1}, {'name': 'Vuur', 'species': 1, 'device': 'Kitchen', 'product_id': 3, 'state': 2}, {'name': 'Vlam', 'species': 1, 'device': None, 'product_id': None, 'state': None}]}
pethub             | Load Devices from pethublocal.db and create Home Assistant MQTT discovery config topics
pethub             | {'name': 'Home hub', 'product_id': 1, 'mac_address': '0000801F121D6719', 'serial_number': 'H004-0088731', 'version': '{"device": {"hardware": 3, "firmware": 2.43}}', 'state': 1, 'battery': '', 'led_mode': 4, 'pairing_mode': 0, 'curfewenabled': None, 'lock_time': None, 'unlock_time': None, 'lockingmode': None, 'bowl1': None, 'bowl2': None, 'bowltarget1': None, 'bowltarget2': None, 'bowltype': None, 'close_delay': None}
pethub             | Loading Hub: Home hub
pethub             | Hub DB record: {'name': 'Home hub', 'product_id': 1, 'mac_address': '0000801F121D6719', 'serial_number': 'H004-0088731', 'version': '{"device": {"hardware": 3, "firmware": 2.43}}', 'state': 1, 'battery': '', 'led_mode': 4, 'pairing_mode': 0, 'curfewenabled': None, 'lock_time': None, 'unlock_time': None, 'lockingmode': None, 'bowl1': None, 'bowl2': None, 'bowltarget1': None, 'bowltarget2': None, 'bowltype': None, 'close_delay': None}
pethub             | Traceback (most recent call last):
pethub             |   File "pethubmqtt.py", line 384, in <module>
pethub             |     states.hub=Box({"State":Online(device.state).name, "Uptime":"0 Mins", "Name": dev, "Serial": device.serial_number,"MAC Address": mac, "LED Mode":Enabled(device.led_mode).name, "Pairing Mode":Enabled(device.pairing_mode).name })
pethub             |   File "/usr/local/lib/python3.8/enum.py", line 339, in __call__
pethub             |     return cls.__new__(cls, value)
pethub             |   File "/usr/local/lib/python3.8/enum.py", line 662, in __new__
pethub             |     raise ve_exc
pethub             | ValueError: 4 is not a valid Enabled
pethub exited with code 1

tinuva · April 23, 2021, 2:41pm

ok I figured it out.

Went into the sqlite db and changed led_state from 4 to 0;
Also had to change pet states from 2 to 1.
And then I also had to add 1 more pet into petstate, which is a new kitten, its learned but never went through the door so I guess I had to fix that for now.

Seems to be running now

peterl · April 23, 2021, 7:07pm

Found the issue David and corrected it, seems I had the hub LED / ears values around the wrong way and hadn’t tested “Dimmed” which is 4. Fixed.

kimagure · April 24, 2021, 9:02am

For catflaps, the pet_* mqtt topics (including ha conf) are not initialized at startup in pethubmqtt.py

Replacing
if pet.product_id == 3: #Pet Door
by
if pet.product_id == 3 or pet.product_id == 6: #Pet Door or Cat Flap
does the trick!

peterl · April 25, 2021, 12:26pm

Little update to the web/app.py to backup the hub firmware just in case and moved the surepet IP address into the config.ini. Thought that might be useful.