If has to be something in the shelve library - we have proven that the state changes are being received …
Only thing I can think of is to reimplement something like shelve in the app.
I also think it’s either shelve or the underlying dbm module.
I just replaced shelve with dbm.dumb and at least the writes seem to work now.
I wanted to try dbm.gnu as this would be an ideal replacement, but trying to import it throws an error; probalby something is missing in the Docker container:
2016-10-19 14:17:38.043684 WARNING ------------------------------------------------------------
2016-10-19 14:17:38.044004 WARNING Unexpected error during loading of switch_reset.py:
2016-10-19 14:17:38.044179 WARNING ------------------------------------------------------------
2016-10-19 14:17:38.066928 WARNING Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/appdaemon/appdaemon.py", line 665, in readApp
importlib.reload(conf.modules[module_name])
File "/usr/local/lib/python3.5/importlib/__init__.py", line 166, in reload
_bootstrap._exec(spec, module)
File "<frozen importlib._bootstrap>", line 626, in _exec
File "<frozen importlib._bootstrap_external>", line 665, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/conf/.apps/switch_reset.py", line 3, in <module>
import dbm.gnu
File "/usr/local/lib/python3.5/dbm/gnu.py", line 3, in <module>
from _gdbm import *
ImportError: No module named '_gdbm'
I need to look into this.
Anyhow, there’s now an error appaerently because the data inside the db is not serialized objects:
2016-10-19 14:27:22.066047 WARNING ------------------------------------------------------------
2016-10-19 14:27:22.069560 WARNING Unexpected error:
2016-10-19 14:27:22.072037 WARNING ------------------------------------------------------------
2016-10-19 14:27:22.137853 WARNING Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/appdaemon/appdaemon.py", line 418, in worker
function(ha.sanitize_timer_kwargs(args["kwargs"]))
File "/conf/.apps/switch_reset.py", line 60, in set_switches
new_state = self.set_state(entity, state = self.device_db[entity])
File "/usr/local/lib/python3.5/site-packages/appdaemon/appapi.py", line 164, in set_state
r = requests.post(apiurl, headers=headers, json=args, verify = conf.certpath)
File "/usr/local/lib/python3.5/site-packages/requests/api.py", line 110, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.5/site-packages/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 461, in request
prep = self.prepare_request(req)
File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 394, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/local/lib/python3.5/site-packages/requests/models.py", line 297, in prepare
self.prepare_body(data, files, json)
File "/usr/local/lib/python3.5/site-packages/requests/models.py", line 428, in prepare_body
body = complexjson.dumps(json)
File "/usr/local/lib/python3.5/json/__init__.py", line 230, in dumps
return _default_encoder.encode(obj)
File "/usr/local/lib/python3.5/json/encoder.py", line 198, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.5/json/encoder.py", line 256, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.5/json/encoder.py", line 179, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'16.0' is not JSON serializable
But I think we’re on the right path…
Still wondering why shelve behaves this way, even when “upgrading” from a python 3.4 docker image to 3.5.
Sebastian
I’m not doing anything special with the underlying DBM so it is probably using whatever I have installed, which could explain differences across the various platforms/images. Also, my understanding is that shelve is meant to serialize before it writes to DBM … but definitely progress! I’ll see if I can figure out which DBM module I am using.
EDIT:
Found this on Stack Overflow:
I think there is no way to specify the underlaying database yourself. shelve uses anydbm and anydbm uses the whichdb module which tries the following underlaying implementations in the following order
dbhash
gdm
dbm
dumbdbm
And by elimination it seems I am using DBM:
hass@Pegasus:~$ python3
Python 3.5.2 (default, Jul 5 2016, 12:43:10)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbhash
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'dbhash'
>>> import gdm
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'gdm'
>>> import dbm
>>>
To my understanding, shelve just wraps up two components: Pickle to serialize objects and dbm.* to store them in a database.
What kind of database depends on the available/installed dbm modules.
You should be able to find the files inside the python search path.
There should be a directory ‘dbm’ containing a file for each db type.
I just changed the Docker base image to python3.6, so the stuff (inside the container) is in:
# ls -l /usr/local/lib/python3.6/dbm/
total 28
-rw-r--r-- 1 root staff 5783 Oct 12 20:13 __init__.py
drwxr-sr-x 2 root staff 113 Oct 19 12:41 __pycache__
-rw-r--r-- 1 root staff 11841 Oct 12 20:13 dumb.py
-rw-r--r-- 1 root staff 72 Oct 12 20:13 gnu.py
-rw-r--r-- 1 root staff 70 Oct 12 20:13 ndbm.py
For each file, you should be able to do an import dbm.(gnu|dumb|ndbm) and use this specific db type.
Since the switch_reset.py only stores basic types (strings, ints, floats…) for each entity, it should not actually be necessary to store serialized objects - instead it should work to just put something like entity_name:'state' into the db. At least that’s what I assumed.
So my thought was to just replace shelve with one of the dbm types, skipping the serialization part, but apparently something broke… I still have to look into that.
Have you been able to reproduce the issue with the appdaemon Docker image?
Using that, we should at least have the same base.
Sebastian
Haven’t tried yet, but I’ll take a look.
Ok, so I now build the appdaemon Docker image based on the python:3.4-alpine image.
Apparently the only usable dbm module there is dbm.dumb, so shelve is automatically using this as db backend.
With this, data storage finally works as expected.
Dbm.dumb might not be the best choice as db backend, though, so I only consider this a workaround.
But as a conclusion I’d say there’s something wrong with whichever dbm module is chosen.
Also Shelve and Pickle don’t seem to be culprits.
@aimc, since it is working for your setup, could you test which dbm module is used in your case?
# python3
Python 3.4.3 (default, Aug 9 2016, 17:10:39)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbm
>>> from dbm import *
>>> dbm.whichdb('switches')
'dbm.dumb'
This should give you the type of the db file (“switches” in my case) that’s used on your system.
Sebastian
Interesting, I am using gnu apparently:
Python 3.5.2 (default, Jul 5 2016, 12:43:10)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbm
>>> from dbm import *
>>> dbm.whichdb('/etc/appdaemon/switches.db')
'dbm.gnu'
>>>
Your guys are talking stuff way over my head but for completeness, on the OIA installer I have below:
Ok, so it looks like dbm.gnu would work and dbm.ndbm won’t.
So I need to figure out how to get dbm.gnu working - this might be a compile-time option for Python.
@Tyfoon: am I correct to assume that you get the same error when you try to import the dbm.gnu module in Python:
# python3
Python 3.4.5 (default, Sep 24 2016, 05:29:30)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbm.gnu
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/dbm/gnu.py", line 3, in <module>
from _gdbm import *
ImportError: No module named '_gdbm'
Can you install the python3-gdbm package as the message suggests?
I don’t know the AIO installer, but try apt-get install python3-gdbm and see if that works.
@aimc: I found a workaround for the Docker users out there.
For some reason the official Python Docker image does not include gnu dbm support.
I’m not sure if there’s a specific reason, but this looks like a bug to me.
So I modified your appdaemon Dockerfile to install the Debian python3-gdbm package (+ dependencies) and then copy the missing library over to the image’s “official” python installation in /usr/local/python3.4.
It looks like this works. Switch_reset now uses a dbm.gnu file to store states and this is updated immediately after changing one of the observed controls.
Here’s my version of the Dockerfile:
FROM python:3.4
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
VOLUME /conf
# Copy source
COPY . .
# INSTALL
RUN apt-get update
RUN apt-get -y install python3-gdbm
RUN cp /usr/lib/python3.4/lib-dynload/_gdbm.cpython-34m-x86_64-linux-gnu.so /usr/local/lib/python3.4/lib-dynload/_gdbm.cpython-34m.so
RUN pip3 install .
CMD [ "appdaemon", "-c", "/conf/appdaemon.cfg" ]
Still, this is a crude hack and it might break at some point.
The best solution would be to get dbm.gnu support fixed in the official Python docker image(s).
Sebastian
Ok guys.
The good news: Running sudo apt-get install python3-gdbm installed the right package and all is up and running as it should! States write to the file and are restored fine!
The bad news: not any so far (just hoping I did not break anything else)
Great news!
This should not break anything in your case because it installs the gdbm module that matches the rest of your Python installation.
Sebastian
Great! Looks like we figured it out - thanks for all your help, I’ll add tha change to th dockefile in the next version.
I also reported it to the python folks:
https://github.com/docker-library/python/issues/151
Let’s see if it gets fixed anytime soon.
Sebastian
1 Like
@sebk-666 Not to necro this, but as I was looking around, I found this and we were discussing it in the Gitter chat.
Can you enlighten me more on how you went about using the MAC Address of the Mac to get OwnTracks to track it when on wifi? I get the concept of the MQTT push but did you end upw riting a PY Script or is there a simpler way?
I wrote a small shell script (I called it ot_ping.sh for “owntracks ping”) that uses the mosquitto_pub tool that comes with the mosquitto mqtt server.
#!/bin/sh
TST=`date +%s`
BATT=`pmset -g batt | grep '%' | awk '{ gsub("%;", ""); print$3 }'`
MSG="{\"t\":\"p\",\"tst\":$TST,\"acc\":1,\"_type\":\"location\",\"alt\":123,\"lon\":1.2345,\"vac\":10,\"lat\":1.2345,\"batt\":$BATT,\"tid\":\"MB\" }"
mosquitto_pub -h <HOST> -p 1883 -u <USER> -P <PASSWORD> -t owntracks/probook/probook -m "$MSG"
This script is called from my crontab every 5 minutes, if I can see the MAC address of my WiFi access point (= when I’m in my local network):
$ crontab -l
*/5 * * * * /sbin/ping -c1 WIFI_AP_IP >/dev/null; if [ "`/usr/sbin/arp -a | /usr/bin/grep 'WIFI_AP_MAC'`" != "" ]; then /path/to/ot_ping.sh; fi
I originally planned to use a tool that could execute arbitrary scripts when a wake-up event is detected (like sleepwatcher).
There exist a few of those tools, but I didn’t come around to test them yet.
The idea behind it all is to turn on the lamp on my desktop whenever I wake my laptop from sleep (and it’s sufficiently dark).
Sebastian
Hi, I am absolutely new to HA and tried to start with scanning my home network with nmap. I have installed HA in a container on my Synology. My question is how do I get rid of the dublicated entries in known-devices. With every scan the entries there are doublicated. Can I solve this by running the “apt-get install python3-gdbm”? How do I modify that command for the docker container?
Thxs and br, Herbert
Hi Herbert,
in my case, the duplicate entries were caused by appdaemon’s switch_reset app.
Are you using appdaemon - you’d definitely know when you are.
Since you’re “absolutely new to HA”, I have my doubts here…
If you aren’t using appdaemon, the problem must be caused by something else.
Sebastian