Local Deployment for SureFlap / SurePetCare Connect using only local MQTT Broker

tinuva · April 5, 2021, 6:23am

Awesome thank you.

Yeah I figured out paho-mqtt on my own. Haven’t updated to the last commit yet, but I don’t see the hub connecting to my local stack.

I can see it is lookup the the DNS:

So 192.168.241.236 is my hub, and if I hover over the response ? I can see it responded with the IP I am pointing the dns name to.

But I don’t see the connection on the http log. I do see when I test with curl. Took me a while to figure out how to make traefik do tcp forwarding (so hope it works). Unless the hub doesn’t support working with SNI then I really have to move it I think.

# curl https://hub.api.surehub.io/ -k -v
*   Trying 192.168.241.12...
* TCP_NODELAY set
* Connected to hub.api.surehub.io (192.168.241.12) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=hub.api.surehub.io
*  start date: Jan 18 08:52:09 2021 GMT
*  expire date: Jan  1 08:52:09 2032 GMT
*  issuer: CN=hub.api.surehub.io
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET / HTTP/1.1
> Host: hub.api.surehub.io
> User-Agent: curl/7.58.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/1.1 200 OK
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< Content-Type: text/html; charset=utf-8
< Content-Length: 37
< Server: Werkzeug/1.0.1 Python/3.7.10
< Date: Mon, 05 Apr 2021 06:18:53 GMT
< 
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection #0 to host hub.api.surehub.io left intact

peterl · April 5, 2021, 7:41am

The hub doesn’t send SNI when it attempts to connect, so the docker compose needs to listen on a real IP not run through a virtual service.
I added a second IP address on my Pi running it and made sure that any existing web servers I had listening on 443 were bound to the main IP.
Then in your docker compose comment out 443 and uncomment the binding with the IP.

#Flask python web server that listens on port 443 for the hub to connect to on boot, also downloads the creds on the first attempt from the cloud service.
  web:
    build: web
    container_name: local_web
    volumes:
      - './output/web:/web/creds'
    ports:
      - "192.168.20.241:443:443"
#      - "443:443"

If it is working you will see something like:

Post payload : {"serial_number": "H0xx-xx", "mac_address": "xx", "product_id": "1", "firmware_version": "2.43"}
192.168.1.167 - - [03/Mar/2021 07:13:59] "POST /api/credentials HTTP/1.1" 200 -

So I think keep it simple and either shutdown all listening services on 443/8883 on the main host, or add a secondary IP to the host and bind to that.

peterl · April 5, 2021, 10:00am

Added support for pet door state as a switch rather than a sensor. Seems like HA MQTT Discovery doesn’t support value_templates too well so I couldn’t have a single state / command value and figure out the switches accordingly. So now have an inbound and outbound switch to display state. Tomorrow I will add the set.

tinuva · April 5, 2021, 1:50pm

ok not sure what I am missing now.

I don’t see anything pushed into HA, but I do see messages logged.

This is what pethub is doing:

local_pethub       | Starting Pet Hub
local_pethub       | Connecting to Home Assistant MQTT endpoint at comms.heaven.za.net port 1883
local_pethub       | {'devices': [('Home hub', 1, '0000801F121D6719', 4, 0, None, None, None, None, None, None, None, None), ('Kitchen', 3, 'D98122FEFFEC1054', None, None, 0, '12:01', '12:02', 2, None, None, None, None)], 'pets': [('Ziva', 2, 'Kitchen', 3, 1), ('Jessie', 2, 'Kitchen', 3, 1)]}
local_pethub       | Load Devices from pethublocal.db and init Home Assistant MQTT discovery configuration
local_pethub       | Loading Hub:  ('Home hub', 1, '0000801F121D6719', 4, 0, None, None, None, None, None, None, None, None)
local_pethub       | Loading Pet Door:  ('Kitchen', 3, 'D98122FEFFEC1054', None, None, 0, '12:01', '12:02', 2, None, None, None, None)
local_pethub       | data published  1
local_pethub       | data published  2
local_pethub       | data published  3
local_pethub       | data published  4
local_pethub       | Door state  2
local_pethub       | data published  5
local_pethub       | data published  6
local_pethub       | data published  7
local_pethub       | data published  8
local_pethub       | Load Pets from pethublocal.db and init Home Assistant MQTT discovery configuration
local_pethub       | Loading Pet: Ziva for feeder Kitchen
local_pethub       | data published  9
local_pethub       | data published  10
local_pethub       | Loading Pet: Jessie for feeder Kitchen
local_pethub       | data published  11
local_pethub       | data published  12
local_pethub       | Subscribe to pethublocal and home assistant topics
local_pethub       | OnMessage pethublocal/messages/D98122FEFFEC1054 1 b'606b141a 07c0 132 117 321 6 01 0c 01 0c 02 01'
local_pethub       | OnMessage pethublocal/messages/D98122FEFFEC1054 1 b'606b141a 07d0 132 118 36 1 04'
local_pethub       | OnMessage pethublocal/messages/D98122FEFFEC1054 1 b'606b141b 07e0 132 119 40 1 02'
local_pethub       | OnMessage pethublocal/messages/D98122FEFFEC1054 1 b'606b141b 07f0 132 120 519 6 02 0c 01 0c 02 01'
local_pethub       | OnMessage pethublocal/messages/D98122FEFFEC1054 1 b'606b141e 0800 132 121 519 6 01 0c 01 0c 02 01'
local_pethub       | OnMessage pethublocal/messages/D98122FEFFEC1054 1 b'606b141e 0810 132 122 36 1 00'
local_pethub       | OnMessage pethublocal/messages/D98122FEFFEC1054 1 b'606b141e 0820 132 123 40 1 02'

This is msgs:

local_msgs         | Apr 05 15:28:47 pethublocal/messages/D98122FEFFEC1054 606b108c 0650 132 99 483 6 01 0b 2d 0c 03 01
local_msgs         | Apr 05 15:28:47 pethublocal/messages/D98122FEFFEC1054 606b108c 0660 132 100 489 6 01 0b 2d 0c 03 01
local_msgs         | Apr 05 15:28:47 pethublocal/messages/D98122FEFFEC1054 606b108c 0670 132 101 495 6 01 0b 2d 0c 03 01
local_msgs         | Apr 05 15:28:47 pethublocal/messages/D98122FEFFEC1054 606b108c 0680 132 102 501 6 01 0b 2d 0c 03 01
local_msgs         | Apr 05 15:28:47 pethublocal/messages/D98122FEFFEC1054 606b108c 0690 132 103 507 6 01 0b 2d 0c 03 01
local_msgs         | Apr 05 15:28:47 pethublocal/messages/D98122FEFFEC1054 606b108c 06a0 132 104 513 6 01 0b 2d 0c 03 01
local_msgs         | Apr 05 15:28:47 pethublocal/messages/D98122FEFFEC1054 606b108c 06b0 132 105 519 6 01 0c 01 0c 02 01
local_msgs         | Apr 05 15:28:47 pethublocal/messages/D98122FEFFEC1054 606b108d 06c0 132 106 632 16 bc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
local_msgs         | Apr 05 15:29:11 pethublocal/messages/D98122FEFFEC1054 606b10a4 06d0 132 107 33 3 95 0c 81
local_msgs         | Apr 05 15:34:00 pethublocal/messages/D98122FEFFEC1054 606b11c5 06e0 8 20 0c 05 b2 00 00 00 00 00 00 00 01 4c
local_msgs         | Apr 05 15:34:00 pethublocal/messages/D98122FEFFEC1054 606b11c5 06f0 132 108 621 9 0c 05 b2 00 00 00 00 00 00
local_msgs         | Apr 05 15:34:57 pethublocal/messages 606b11fe 0700 132 109 54 3 03 4f 44
local_msgs         | Apr 05 15:35:02 pethublocal/messages 606b1203 0710 132 110 54 3 02 50 41
local_msgs         | Apr 05 15:35:08 pethublocal/messages/D98122FEFFEC1054 606b120c 0720 8 00 0c 06 61 00 0f 01 44 01 01 01 01 5d
local_msgs         | Apr 05 15:35:08 pethublocal/messages/D98122FEFFEC1054 606b120c 0730 132 111 525 3 0c 06 61
local_msgs         | Apr 05 15:35:19 pethublocal/messages/D98122FEFFEC1054 606b1217 0740 8 20 0c 07 b5 00 00 00 00 00 00 02 01 5f
local_msgs         | Apr 05 15:35:19 pethublocal/messages/D98122FEFFEC1054 606b1217 0750 132 112 621 9 0c 07 b5 00 00 00 00 00 00
local_msgs         | Apr 05 15:38:10 pethublocal/messages 606b12c2 0760 132 113 54 1 03
local_msgs         | Apr 05 15:38:10 pethublocal/messages 606b12c2 0770 132 114 56 1 45
local_msgs         | Apr 05 15:38:15 pethublocal/messages/D98122FEFFEC1054 606b12c7 0780 8 01 0c 0a 61 00 02 03 44 03 03 03 01 87
local_msgs         | Apr 05 15:38:15 pethublocal/messages 606b12c7 0790 132 115 54 3 02 51 42
local_msgs         | Apr 05 15:38:15 pethublocal/messages/D98122FEFFEC1054 606b12c7 07a0 8 01 0c 0a 81 00 0f 04 44 04 04 04 01 88
local_msgs         | Apr 05 15:38:15 pethublocal/messages/D98122FEFFEC1054 606b12c7 07b0 132 116 528 3 0c 0a 81
local_msgs         | Apr 05 15:43:54 pethublocal/messages/D98122FEFFEC1054 606b141a 07c0 132 117 321 6 01 0c 01 0c 02 01
local_msgs         | Apr 05 15:43:54 pethublocal/messages/D98122FEFFEC1054 606b141a 07d0 132 118 36 1 04
local_msgs         | Apr 05 15:43:55 pethublocal/messages/D98122FEFFEC1054 606b141b 07e0 132 119 40 1 02
local_msgs         | Apr 05 15:43:55 pethublocal/messages/D98122FEFFEC1054 606b141b 07f0 132 120 519 6 02 0c 01 0c 02 01
local_msgs         | Apr 05 15:43:58 pethublocal/messages/D98122FEFFEC1054 606b141e 0800 132 121 519 6 01 0c 01 0c 02 01
local_msgs         | Apr 05 15:43:58 pethublocal/messages/D98122FEFFEC1054 606b141e 0810 132 122 36 1 00
local_msgs         | Apr 05 15:43:59 pethublocal/messages/D98122FEFFEC1054 606b141e 0820 132 123 40 1 02

Actually looking at it again…it thinks I have a pet feeder? But I only have a pet door and 2 pets on the door currently.

The DB looks ok though…

sqlite> select * from devices;
0000801F121D6719|1|Home hub|H004-0088731||||{"device": {"hardware": 3, "firmware": 2.43}}
D98122FEFFEC1054|3|Kitchen||4.9566|-70.16666666666667|-73.8|{"lcd": {"hardware": 1, "firmware": 1}, "rf": {"hardware": 4, "firmware": 0.16}}
sqlite> select * from doors;
D98122FEFFEC1054|0|12:01|12:02|2
sqlite> select * from feeders;
sqlite> select * from hubs;
0000801F121D6719|4|0
sqlite> select * from pets;
711033360D00|Ziva|2

peterl · April 5, 2021, 10:56pm

Humm this shouldn’t be happening:

local_pethub       | OnMessage pethublocal/messages/D98122FEFFEC1054 1 b'606b141a 07c0 132 117 321 6 01 0c 01 0c 02 01'

As OnMessage means that it isn’t calling the petdoor function, so I need to review my code this evening to see why that is happening as it should be calling on_petdoor_message but it doesn’t seem to be.

And this was just a typo where I got the door and feeder around the wrong way, it is loading it for the door.

local_pethub       | Loading Pet: Ziva for <s>feeder</s> pet door Kitchen
local_pethub       | Loading Pet: Jessie for <s>feeder</s> pet door Kitchen

tinuva · April 6, 2021, 7:00am

ok great, so its working mostly.

This morning, some of the sensors started showing up. Like:

pet door as name Kitchen
unlock and lock time
curfew
pets as their names etc

Maybe a nice improvement will be to add the following to the home assistant auto configuration:

unique_id:
An ID that uniquely identifies this sensor. If two sensors have the same unique ID, Home Assistant will raise an exception.

device:
You must have the same parameters for each entity

So for device it can be the device serial/nr as in the sqlite db
Unique id per mqtt item can be soemthing like device+sensor name.

You can see how someone else suggested this to me on something else I worked on at MQTT demultiplexer automatic sensor discovery for tasmota2zigbee (last post currently)

Then it will be much easier to find the sensors in the mqtt integration, I currently have MANY sensors in there, so difficult to figure out which ones are from local_pethub.

peterl · April 6, 2021, 7:06am

I also realised how I messed up but didn’t find it as my config doesn’t update from default. The web/app.py creates the topic name as pethublocal so the hub will be creating messages in pethublocal/messages whereas in the mqtt/default/conf/mosquitto.conf I had pethublocal/hub/ as I originally had pethublocal/hub/messages expecting that I would be creating topics like pethublocal/local/door
Just corrected them all locally and going to push them once I have fully tested it.
Like the suggestion of using the uniqueid as the mac/tagid. I went for the name so that human readable values were the topic names, but as you say it won’t work for everyone… let me thing about it… and patches welcome.

tinuva · April 6, 2021, 7:25am

Ah now I know why the one mosquitto didn’t forward to the main one. I eventually added the certificates to my main mosquitto container and had everything connect directly to that.

peterl · April 6, 2021, 10:41am

Just added support for setting the lock state inbound and outbound on the door via the inbound and outbound switches.

Seems to work for me ™

tinuva · April 6, 2021, 11:45am

I get this:

local_pethub       | Traceback (most recent call last):
local_pethub       |   File "pethubmqtt.py", line 145, in <module>
local_pethub       |     ret=mc.publish(device_switch_t+devl+"_"+key+'/config',json.dumps(configmessage))
local_pethub       | NameError: name 'key' is not defined

Do we need key here? Its not the same as the switch.

tinuva · April 6, 2021, 12:26pm

Well my pet door lock and unlock works woohoo

kimagure · April 6, 2021, 4:31pm

Good news!

I hope we can have the same thing with the cat flap soon

@peterl, I sent you a few frames I captured with my CC2531 sniffer yesterday.
Do not hesitate to tell me if you need more (or something different).

Thanks

K.

peterl · April 6, 2021, 5:14pm

Support for locking the cat door is on my todo for tonight if time permits around the family.

tinuva · April 7, 2021, 7:14am

@peterl

Some thoughts after running this for a while.

Probably need to look at resending (publishing) the home assistant auto config messages to mqtt on a regular interval. Running local_pethub for a few hours, my sensors in home assistant go unavailable. Restarting local_pethub fixes this.
Would be nice if the settings are read from environment variables. That will allow one to put the settings like the ip address of the mqtt server in a .env file, same folder as docker-compose.yaml which can be added to .gitignore. Will be much easier each time to apply updates from github for users.
I can think of a few architecture changes going forward, to make this easier and simpler. Like it would be nice to have a webui to configure pets and hub names in the sqlite db. Another option is, to combine the mkdb part with local_pethub, and then if the .db file doesnt exist but the username/password exist in environment variables then create the db before launching the mqtt processor. Kinda like a first run setup. But I am guessing most of this will be long term ideas after most of the functionality for the messages (like cat flap) is figured out and working.

peterl · April 7, 2021, 9:22am

Completely agree with everything you’ve said and variables was high up on my priority but I have just worked out the cat door locking and it has meant I had to deal with the feeder/flap counter issue where each message to the feeder and cat flap has a counter… so need to store and iterate that.
So database change unfortunately and I think I might refactor to get rid of zigsniff and move that to the to folder and have local at the docker level. Then add a script to start it pulling in an env file. Time is short and the list of things to add is long.

peterl · April 7, 2021, 10:20am

Have the cat flap send logic sorted, just need to get the current state in… but this is what I have figured out.
Going to commit the code which may break stuff, but off to bed now.

    elif devicetype=="catflap":
        if operation == "dumpstate": #Dump all memory registers from 0 to 630
            msgstr = "TBC"
        if operation == "inbound" or operation == "outbound":
            curs.execute('select mac_address, lockingmode from devices join doors using (mac_address) where name like (?)', ([device]))
            lockingmode = curs.fetchone()
            #print("Current locking mode: " + lockingmode[0])
            if (operation == "outbound" and state == "OFF" and lockingmode[1] == 1) or (operation == "inbound" and state == "OFF" and lockingmode[1] == 2):
                #Going to Lock State 0 - Unlocked
                msgstr = "11 00 CC 00 TT TT TT TT 00 00 00 00 00 00 07 06 00 02"
            elif (operation == "outbound" and state == "ON" and lockingmode[1] == 0) or (operation == "inbound" and state == "OFF" and lockingmode[1] == 3):
                #Going to Lock State 1 - Keep pets in
                msgstr = "11 00 CC 00 TT TT TT TT 00 00 00 00 00 00 07 03 00 02"
            elif (operation == "outbound" and state == "OFF" and lockingmode[1] == 3) or (operation == "inbound" and state == "ON" and lockingmode[1] == 0):
                #Going to Lock State 2 - Keep pets out
                msgstr = "11 00 CC 00 TT TT TT TT 00 00 00 00 00 00 07 05 00 02"
            elif (operation == "outbound" and state == "ON" and lockingmode[1] == 2) or (operation == "inbound" and state == "ON" and lockingmode[1] == 1):
                #Going to Lock State 3 - Lock both ways
                msgstr = "11 00 CC 00 TT TT TT TT 00 00 00 00 00 00 07 04 00 02"
            else:
                msgstr = "11 00 CC 00 TT TT TT TT 00 00 00 00 00 00 07 06 00 02"
            hubts = feedertimestampfromnow()
            devcounter = devicecounter(lockingmode[0],"-1","-2") #Iterate the send counter for the device
            msgstr = msgstr.replace('CC', hb(devcounter['send'])) # Replace device counter in the record
            msgstr = msgstr.replace('TT TT TT TT', " ".join(hubts[i:i+2] for i in range(0, len(hubts), 2))) # Timestamp
            return {"topic":"pethublocal/messages/"+lockingmode[0], "msg":buildmqttsendmessage(msgstr)}
        return buildmqttsendmessage(msgstr)

Need a new table:

CREATE TABLE devicecounter(mac_address TEXT, send INTEGER, retrieve INTEGER );
INSERT INTO devicecounter VALUES('macaddy',0,0);

kimagure · April 7, 2021, 10:24am

Thanks! I will try this and will give you some feedback.

K.

peterl · April 7, 2021, 10:35am

I haven’t yet added everything into pethubmqtt.py yet so it won’t work as it won’t create the topics required for listening and I haven’t mapped the incoming messages to the mqtt topics for status responses. But you could try poking the messages to the door directly using a send command:

import pethubpacket as p
import paho.mqtt.client as mqtt

resp = p.generatemessage("catflap", "Kitchen", "outbound", "ON")
print(resp)

mqtthost = 'hubmq'
mqttport = 1883

mc = mqtt.Client()
mc.connect(mqtthost, mqttport, 30)

ret=mc.publish(resp['topic'],resp['msg'],qos=1)
print(ret)

That should work in the pethub directory with your db and associated bits.
Edit. Just pushed to include the 127 message type prefix… should look good now:

python3 send.py
{'topic': 'pethublocal/messages/xx', 'msg': '606d57f7 1000 127 11 00 07 00 7a 6e 10 55 00 00 00 00 00 00 07 06 00 02'}

kimagure · April 7, 2021, 8:52pm

@peterl, if you implement this, please also add environment variables for user/password authentication on the main mqtt broker.

The credentials can be passed to the paho-mqtt connection using username_pw_set(username="xxx",password="xxx") and will have to be added to the mosquitto.conf file as well (remote_username/remote_password).

With all variables set in the .env file, it’s perfect!

Impressive job anyway! Thanks!

K.

peterl · April 7, 2021, 11:55pm

If time permits over the next few days I will look to do some refactoring and move the following around as that makes sense if I clean things up and improve the deployment process.

/docker/zigsniff -> /zigsniff
/docker/local/*-> /docker
/PetHub -> /docker/source
Change the compose containers to use a shared volume of /pethub/source mapped to /docker/source that way you don’t need to keep on copying the files in
Add /docker/.env environmental file and then apply using those variables across the containers

I will also look to collapse web and pethub containers into a single container so then a single web flask + paho-mqtt container on 80 & 443 so it can service the hub credentials request and be a web server for end users to configure the database and configure infrequent activity that doesn’t logically sit or easily work within home assistant such as the device adoption process and managing provisioned pets on the devices… that will be a longer term activity once the core functionality is sorted.