Dahua IPC to MQTT App

[ReneTode]

i found this on the pycurl github:

The line of code that throws Could not run curl-config: [Errno 2] No such file or directory” when installing pycurl on Linux, should inform the user as to remedy - a pre-requisite, rather than leaving them to test their google skills. A page online is the remedy, and it could have sections like the following for Raspbian GNU/Linux 8 (jessie):

sudo apt-get install libssl-dev libcurl4-openssl-dev python-dev

i guess they didnt succeed in preventing the google hunt.
so the search goes on to the right packages.

libssl-dev
libcurl4-openssl-dev

could do the trick as system packages.

and i did some searching on egg-info errors, and it seems thats a part from the python package setuptools

so its probably to much installing, but could do the trick (remove what throws an error)

{
  "log_level": "info",
  "system_packages": [
    "libcurl",
    "python3-dev",
    "libssl-dev",
    "libcurl4-openssl-dev"
  ],
  "python_packages": [
    "pip==19.0.2",
    "setuptools",
    "pycurl"
  ]
}

[algirdasc (Algirdas Č)]

algirdas@devlaptop:~$ apt-file search curl-config | grep bin/curl-config
libcurl-openssl1.0-dev: /usr/bin/curl-config
libcurl4-gnutls-dev: /usr/bin/curl-config
libcurl4-nss-dev: /usr/bin/curl-config
**libcurl4-openssl-dev: /usr/bin/curl-config**

I guess the last one is missing.

[xbmcnut]

@algirdasc

I guess the last one is missing.

Yep, didn’t like that.

OK: 22225 distinct packages available
OK: 73 MiB in 33 packages
(1/2) Installing pkgconf (1.5.3-r0)
(2/2) Installing python3-dev (3.6.6-r0)
Executing busybox-1.28.4-r1.trigger
OK: 91 MiB in 35 packages
  libssl-dev (missing):
ERROR: unsatisfiable constraints:
    required by: world[libssl-dev]
FATAL: Failed installing package libssl-dev

[ReneTode]

oke, try again without libssl-dev

[xbmcnut]

Had to remove that and libcurl4-openssl-dev. Sadly, same errors. I’m sure there will be some combo that works?

{
  "log_level": "info",
  "system_packages": [
    "libcurl",
    "python3-dev"
  ],
  "python_packages": [
    "pip==19.0.2",
    "setuptools",
    "pycurl"
  ]
}

Successfully installed pip-19.0.2
Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages (39.0.1)
Collecting pycurl
  Downloading https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz (214kB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/tmp/pip-install-jpa84dl3/pycurl/setup.py", line 223, in configure_unix
        stdout=subprocess.PIPE, stderr=subprocess.PIPE)
      File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
        restore_signals, start_new_session)
      File "/usr/lib/python3.6/subprocess.py", line 1344, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'curl-config': 'curl-config'

[algirdasc (Algirdas Č)]

Do you have access to shell via SSH or directly?

[johnnyletrois]

I use this python script to monitor Dahua IVS events and it works great for me:

Dahua IP cam IVS event as binary_sensor equals smarter camera motion detection

The Dahua Starlight 5231 IP cam is one of the best bang for the buck POE IP cameras on the market. See extensive review and discussion here: https://ipcamtalk.com/threads/dahua-starlight-varifocal-turret-ipc-hdw5231r-z.14683/ Camera-based motion detection is pretty inaccurate and ends in lots of false positives. New cameras from Hikvision and Dahua and other manufacturers include smarter motion detection with virtual tripwires, zone intrusion, etc. that is significantly more accurate. User c…

[ReneTode]

i think the only thing that is missing is a package that includes curl-config and can be installed on hassio.
the problem is to find that package.

these packages seem to contain that:

* libcurl4-gnutls-dev
* libcurl4-nss-dev
* libcurl4-openssl-dev

libflickcurl-dev
curl-dev

its very well possible that curl-dev now works, now that we found out the right pip and the need for python-dev

@algirdasc there is no use trying trough SSH.
the package needs to be installed inside the docker

[algirdasc (Algirdas Č)]

I had to enter my appdaemon docker container and install packages manually.
apk add curl-dev python3-dev libressl-dev
After that I just created requirements.txt with pycurl in it. But I’m not 100% sure if thats the correct way on HassOS.

[ReneTode]

@algirdasc

I had to enter my appdaemon docker container and install packages manually.

if i have understand apk correct then you dont install with apk add, but you make it available.
the install is done by the requirements file.
but i doubt that they have even apk available in hassio/hassOS/addons.

the problem is that not everything is available for such a slimmed down OS.

[ReneTode]

just an idea (dont know if its possible and if its worth it)
but you use pycurl for the connection, dont you?
isnt the same thing possible with something like requests?

[xbmcnut]

@ReneTode

libflickcurl-dev
curl-dev

Shall I try adding those?

[ReneTode]

curl-dev was already tried as python package, but maybe its a system package
i would try on or the other and remove what doesnt work.

[xbmcnut]

Making some progress. Came up with a ‘gcc’ error so I added that and we move forward but not quite there yet.

{
  "log_level": "info",
  "system_packages": [
    "libcurl",
    "python3-dev",
    "curl-dev",
    "gcc"
  ],
  "python_packages": [
    "pip==19.0.2",
    "pycurl"
  ]
}

OK: 22225 distinct packages available
OK: 73 MiB in 33 packages
(1/2) Installing pkgconf (1.5.3-r0)
(2/2) Installing python3-dev (3.6.6-r0)
Executing busybox-1.28.4-r1.trigger
OK: 91 MiB in 35 packages
(1/6) Upgrading libcurl (7.61.1-r0 -> 7.61.1-r1)
(2/6) Installing libressl-dev (2.7.4-r0)
(3/6) Installing zlib-dev (1.2.11-r1)
(4/6) Installing libssh2-dev (1.8.0-r3)
(5/6) Installing nghttp2-dev (1.32.0-r0)
(6/6) Installing curl-dev (7.61.1-r1)
Executing busybox-1.28.4-r1.trigger
OK: 116 MiB in 40 packages
(1/10) Installing binutils (2.30-r5)
(2/10) Installing gmp (6.1.2-r1)
(3/10) Installing isl (0.18-r0)
(4/10) Installing libgomp (6.4.0-r9)
(5/10) Installing libatomic (6.4.0-r9)
(6/10) Installing libgcc (6.4.0-r9)
(7/10) Installing mpfr3 (3.1.5-r1)
(8/10) Installing mpc1 (1.0.3-r1)
(9/10) Installing libstdc++ (6.4.0-r9)
(10/10) Installing gcc (6.4.0-r9)
Executing busybox-1.28.4-r1.trigger
OK: 184 MiB in 50 packages
[cont-init.d] 80-system-packages.sh: exited 0.
[cont-init.d] 81-python-packages.sh: executing... 
Collecting pip==19.0.2
  Downloading https://files.pythonhosted.org/packages/d7/41/34dd96bd33958e52cb4da2f1bf0818e396514fd4f4725a79199564cd0c20/pip-19.0.2-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
Successfully installed pip-19.0.2
Collecting pycurl
  Downloading https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz (214kB)
Installing collected packages: pycurl
  Running setup.py install for pycurl: started
    Running setup.py install for pycurl: finished with status 'error'
    Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-g17t7os4/pycurl/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-8yszvic9/install-record.txt --single-version-externally-managed --compile:
    Using curl-config (libcurl 7.61.1)
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-aarch64-3.6
    creating build/lib.linux-aarch64-3.6/curl
    copying python/curl/__init__.py -> build/lib.linux-aarch64-3.6/curl
    running build_ext
    building 'pycurl' extension
    creating build/temp.linux-aarch64-3.6
    creating build/temp.linux-aarch64-3.6/src
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -g -Os -g -Os -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DPYCURL_VERSION="7.43.0.2" -DHAVE_CURL_SSL=1 -DHAVE_CURL_OPENSSL=1 -DHAVE_CURL_SSL=1 -I/usr/include/python3.6m -c src/docstrings.c -o build/temp.linux-aarch64-3.6/src/docstrings.o
    In file included from src/pycurl.h:4:0,
                     from src/docstrings.c:4:
    /usr/include/python3.6m/Python.h:11:20: fatal error: limits.h: No such file or directory
     #include <limits.h>
                        ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-g17t7os4/pycurl/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-8yszvic9/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-g17t7os4/pycurl/
FATAL: Failed installing package pycurl
[cont-init.d] 81-python-packages.sh: exited 1.
[cont-finish.d] executing container finish scripts...
[cont-finish.d] 50-compiled-symlink.sh: executing... 
[cont-finish.d] 50-compiled-symlink.sh: exited 0.
[cont-finish.d] 99-message.sh: executing... 
-----------------------------------------------------------
                Oops! Something went wrong.

[xbmcnut]

SUCCESS Peeps!!!

Installing collected packages: pycurl
  Running setup.py install for pycurl: started
    Running setup.py install for pycurl: finished with status 'done'
Successfully installed pycurl-7.43.0.2
[cont-init.d] 81-python-packages.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
2019-02-14 13:02:42.575306 INFO AppDaemon Version 3.0.2 starting
2019-02-14 13:02:42.576080 INFO Configuration read from: /config/appdaemon/appdaemon.yaml
2019-02-14 13:02:42.579838 INFO AppDaemon: Starting Apps
2019-02-14 13:02:42.589278 INFO AppDaemon: Loading Plugin HASS using class HassPlugin from module hassplugin
2019-02-14 13:02:42.873772 INFO AppDaemon: HASS: HASS Plugin Initializing
2019-02-14 13:02:42.875057 INFO AppDaemon: HASS: HASS Plugin initialization complete
2019-02-14 13:02:42.876337 INFO Starting Dashboards
2019-02-14 13:02:42.898533 INFO API is disabled
2019-02-14 13:02:42.915773 INFO AppDaemon: HASS: Connected to Home Assistant 0.87.1
2019-02-14 13:02:43.283329 INFO AppDaemon: Got initial state from namespace default
2019-02-14 13:02:45.141222 INFO AppDaemon: Reading config
2019-02-14 13:02:45.148381 INFO AppDaemon: /config/appdaemon/apps/apps.yaml added or modified
2019-02-14 13:02:45.149005 INFO AppDaemon: /config/appdaemon/apps/apps.yaml added or modified
2019-02-14 13:02:45.149590 INFO AppDaemon: App 'hello_world' added
2019-02-14 13:02:45.150527 INFO AppDaemon: Adding /config/appdaemon/apps to module import path
2019-02-14 13:02:45.152064 INFO AppDaemon: Loading App Module: /config/appdaemon/apps/hello.py
2019-02-14 13:02:45.183009 INFO AppDaemon: Initializing app hello_world using class HelloWorld from module hello
2019-02-14 13:02:45.426174 INFO hello_world: Hello from AppDaemon
2019-02-14 13:02:45.429993 INFO hello_world: You are now ready to run Apps!
2019-02-14 13:02:45.432048 INFO AppDaemon: App initialization complete

{
  "log_level": "info",
  "system_packages": [
    "libcurl",
    "python3-dev",
    "curl-dev",
    "gcc",
    "g++"
  ],
  "python_packages": [
    "pip==19.0.2",
    "pycurl"
  ]
}

Needed latest g++ compiler added. Can remove ‘gcc’ as that gets added anyway

[xbmcnut]

Getting real close!!

Here is my app.yaml file.

---
hello_world:
  module: hello
  class: HelloWorld

dahua_mqtt:
  module: dahua_mqtt
  class: DahuaMQTT

And the error;

2019-02-14 13:14:33.236980 INFO hello_world: You are now ready to run Apps!
2019-02-14 13:14:33.237597 INFO AppDaemon: Initializing app dahua_mqtt using class DahuaMQTT from module dahua_mqtt
2019-02-14 13:14:33.238487 WARNING AppDaemon: ------------------------------------------------------------
2019-02-14 13:14:33.239081 WARNING AppDaemon: Unexpected error running initialize() for dahua_mqtt
2019-02-14 13:14:33.239631 WARNING AppDaemon: ------------------------------------------------------------
2019-02-14 13:14:33.241850 WARNING AppDaemon: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/appdaemon/appdaemon.py", line 1581, in init_object
    init()
  File "/config/appdaemon/apps/dahua_mqtt.py", line 66, in initialize
    for camera in self.args["cameras"]:
KeyError: 'cameras'
2019-02-14 13:14:33.242414 WARNING AppDaemon: ------------------------------------------------------------
2019-02-14 13:14:33.244726 INFO AppDaemon: App initialization complete

Seems there is something in the code that might require customisation??

class DahuaMQTT(hass.Hass):

	proc = None
	cameras = []
	curl_multiobj = pycurl.CurlMulti()
	num_curlobj = 0
	kill_thread = False

	def initialize(self):

		for camera in self.args["cameras"]:
			dahuacam = DahuaCamera(self, camera)
			self.cameras.append(dahuacam)
			url = URL_TEMPLATE.format(**camera)

			curlobj = pycurl.Curl()
			dahuacam.curlobj = curlobj

			curlobj.setopt(pycurl.URL, url)
			curlobj.setopt(pycurl.CONNECTTIMEOUT, 30)
			curlobj.setopt(pycurl.TCP_KEEPALIVE, 1)
			curlobj.setopt(pycurl.TCP_KEEPIDLE, 30)
			curlobj.setopt(pycurl.TCP_KEEPINTVL, 15)
			curlobj.setopt(pycurl.HTTPAUTH, pycurl.HTTPAUTH_DIGEST)
			curlobj.setopt(pycurl.USERPWD, "{0}:{1}".format(camera["user"], camera["pass"]))
			curlobj.setopt(pycurl.WRITEFUNCTION, dahuacam.on_receive)

[ReneTode]

you forgot to configure in app.yaml

Example configuration:
DahuaMQTT:
  class: DahuaMQTT
  module: dahua_mqtt
  cameras:
    - host: 192.168.0.1
      port: 80
      user: user
      pass: pass
      topic: cameras/1
      events: VideoMotion,VideoBlind,VideoLoss,AlarmLocal,....
    - host: 192.168.0.2
      port: 80
      user: user
      pass: pass
      topic: cameras/2
      events: VideoMotion,VideoBlind,VideoLoss,AlarmLocal,....

glad you got pycurl finally installed

[xbmcnut]

@ReneTode

you forgot to configure in app.yaml

I’m a dumb arse!! I totally see the example config in the .py file now I’m looking! Thanks for that.

[xbmcnut]

Seems to be working now but despite wire cross detection configured, no MQTT events. Hmmm…

2019-02-14 13:47:09.485024 INFO AppDaemon: Reading config
2019-02-14 13:47:09.520069 INFO AppDaemon: /config/appdaemon/apps/apps.yaml added or modified
2019-02-14 13:47:09.520747 INFO AppDaemon: App 'DahuaMQTT' changed
2019-02-14 13:47:09.522550 INFO AppDaemon: Terminating DahuaMQTT
2019-02-14 13:47:09.523150 INFO AppDaemon: Calling terminate() for DahuaMQTT
2019-02-14 13:47:09.546637 INFO AppDaemon: Initializing app DahuaMQTT using class DahuaMQTT from module dahua_mqtt
2019-02-14 13:47:09.554993 INFO DahuaMQTT: [10.0.1.11] OnDisconnect(Success [])
2019-02-14 13:47:09.602785 INFO DahuaMQTT: [10.0.1.14] OnConnect()
2019-02-14 13:47:09.609108 INFO DahuaMQTT: [10.0.1.16] OnConnect()
2019-02-14 13:47:13.428498 INFO DahuaMQTT: [10.0.1.11] OnDisconnect(Success [])
2019-02-14 13:47:14.739967 INFO DahuaMQTT: [10.0.1.11] OnDisconnect(Success [])
2019-02-14 13:47:18.558975 INFO DahuaMQTT: [10.0.1.11] OnDisconnect(Success [])

[Velly (Vel Kilmor)]

Tryed too: following log
in MQTT FX topic “cameras/dahua/VideoMotion” (I changed “1” to “dahua” in apps.yaml) gives nothing.
Mqtt discovery is enabled.
Maybe I should change topic for Hassio as “homeassistant/cameras/dahua” ??
#MQTT
mqtt:
broker: 192.168.2.162
discovery: true
discovery_prefix: homeassistant
port: 1883
keepalive: 60
username: user
password: mypass

• 1 question: is that normal that on every HASSIO reboot or AppDaemon restart action , AppDaemon do installation of all mentioned packages again? It takes so much time… Is that normal behaviour?
  And some strange happens in logs - on every few restarts it installs pip 10, and pip 19, then uninstall pip 10 etc etc etc.

    2019-02-14 17:22:48.604907 INFO AppDaemon Version 3.0.2 starting
    2019-02-14 17:22:48.605931 INFO Configuration read from: /config/appdaemon/appdaemon.yaml
    2019-02-14 17:22:48.612680 INFO AppDaemon: Starting Apps
    2019-02-14 17:22:48.626707 INFO AppDaemon: Loading Plugin HASS using class HassPlugin from module hassplugin
    2019-02-14 17:22:49.185514 INFO AppDaemon: HASS: HASS Plugin Initializing
    2019-02-14 17:22:49.187362 INFO AppDaemon: HASS: HASS Plugin initialization complete
    2019-02-14 17:22:49.188969 INFO Starting Dashboards
    2019-02-14 17:22:49.234088 INFO API is disabled
    2019-02-14 17:22:49.288354 INFO AppDaemon: HASS: Connected to Home Assistant 0.86.4
    2019-02-14 17:22:51.018971 INFO AppDaemon: Got initial state from namespace default
    2019-02-14 17:22:52.697659 INFO AppDaemon: Reading config
    2019-02-14 17:22:52.733463 INFO AppDaemon: /config/appdaemon/apps/apps.yaml added or modified
    2019-02-14 17:22:52.734208 INFO AppDaemon: /config/appdaemon/apps/apps.yaml added or modified
    2019-02-14 17:22:52.734912 INFO AppDaemon: App 'hello_world' added
    2019-02-14 17:22:52.735641 INFO AppDaemon: App 'DahuaMQTT' added
    2019-02-14 17:22:52.736712 INFO AppDaemon: Adding /config/appdaemon/apps to module import path
    2019-02-14 17:22:52.739136 INFO AppDaemon: Loading App Module: /config/appdaemon/apps/hello.py
    2019-02-14 17:22:52.807824 INFO AppDaemon: Loading App Module: /config/appdaemon/apps/dahua_mqtt.py
    2019-02-14 17:22:52.848050 INFO AppDaemon: Initializing app hello_world using class HelloWorld from module hello
    2019-02-14 17:22:53.307600 INFO hello_world: Hello from AppDaemon
    2019-02-14 17:22:53.314276 INFO hello_world: You are now ready to run Apps!
    2019-02-14 17:22:53.315391 INFO AppDaemon: Initializing app DahuaMQTT using class DahuaMQTT from module dahua_mqtt
    2019-02-14 17:22:53.322199 INFO AppDaemon: App initialization complete
    2019-02-14 17:22:54.972803 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:22:59.997791 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:23:05.009103 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:23:10.040252 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:23:15.049828 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:23:20.085555 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:23:25.093993 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:23:30.123956 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:23:35.133519 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:23:40.166384 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:23:45.174309 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:23:50.206354 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:23:55.214910 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:00.261864 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:05.287968 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:10.297348 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:15.330974 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:20.378761 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:25.413533 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:30.424347 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:35.451889 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:40.461140 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:45.490766 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:50.500577 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:24:55.509265 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:25:00.534669 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])
    2019-02-14 17:25:05.547380 INFO DahuaMQTT: [192.168.2.172] OnDisconnect(Success [])