Issue upgrading Appdamond to latest version

Avi_Cohen · February 2, 2020, 11:18pm

trying to update appdaemon to latest version and getting tons of errors
and appdaemon not started.
I saved previous appdaemon image so when i change the image to old one
so when i downgrade back - all is working.
I’d like to update appdaemon to latest version but not sure
what to change in my configuration :

docker-compose:

appdaemon:
     container_name: appdaemon
     image: avico78/appdaemon
     environment:
       - HA_URL="http://192.168.1.112:8123"
       - token="welcome"
     ports:
       - 5050:5050
     networks:
     - traefik_proxy  #as reverse proxy
     volumes:
       - ${PWD}/appdaemon/config:/conf
     depends_on:
       - homeassistant

in app/requirements.txt

{
  "log_level": "info",
  "system_packages": [
    "libcurl",
    "python-dev"
  ],
  "python_packages": [
    "pip==20.0.2",
    "pycurl"
  ]
}

appdaemon.yaml

log:
  logfile: /conf/appdaemon.log
  errorfile: /conf/aderror.log
  accessfile: /conf/access.log
appdaemon:
  threads: 10
  app_dir: /conf/apps
  plugins:
    HASS:
      type: hass
      ha_url: "http://192.168.1.112:8123"
      token: '123123123.123123.234234-23424'
hadashboard:
  dash_url: http://192.168.1.113:5050
  dash_dir: /conf/dashboards

Burningstone · February 2, 2020, 11:49pm

You need to correct your appdaemon.yaml, the configuration change quite a bit from 3 to 4. Check out my settings in another topic where I post a working appdaemon.yaml + docker-compose

Avi_Cohen · February 13, 2020, 3:02pm

Thanks @Burningstone ,

I update the docker-compose as following:

  appdaemon_4:
      container_name: appdaemon_4
      image: acockburn/appdaemon:latest
      ports:
        - 5050:5050
      networks:
        - traefik_proxy
      volumes:
        - /etc/localtime:/etc/localtime:ro
        - ${PWD}/appdaemon4/config:/conf
      depends_on:
        - homeassistant
      restart: unless-stopped

In appdaemon.yaml

secrets: /conf/secrets.yaml
logs:
  main_log:
    filename: /conf/logs/appdaemon.log
  error_log:
    filename: /conf/logs/error.log
  access_log:
    filename: /conf/logs/access.log
appdaemon:
  time_zone: !secret timezone
  latitude: !secret latitude
  longitude: !secret longitude
  elevation: !secret elevation
  missing_app_warnings: 1
  production_mode: False
  app_dir: /conf/apps
  plugins:
    HASS:
      type: hass
      namespace: hass
      token: !secret appdaemon_token
      ha_url: "http://192.168.1.112:8123"
    MQTT:
      type: mqtt
      namespace: mqtt
      client_id: AD_4
      client_host: !secret mqtt_broker
http:
  url: http://192.168.1.112:5050
api:
hadashboard:
admin:
  title: AppDaemon
  stats_update: realtime

Appdaemon is up and running:

2020-02-13 16:59:24.390235 INFO HASS: Connected to Home Assistant 0.104.3
2020-02-13 16:59:25.506390 INFO HASS: Evaluating startup conditions
2020-02-13 16:59:25.619084 INFO AppDaemon: Got initial state from namespace hass
2020-02-13 16:59:26.685269 INFO AppDaemon: Scheduler running in realtime
2020-02-13 16:59:26.693511 INFO AppDaemon: Adding /conf/apps to module import path
2020-02-13 16:59:26.701070 INFO AppDaemon: Loading App Module: /conf/apps/dahua_mqtt.py
2020-02-13 16:59:26.718158 WARNING AppDaemon: Unexpected error loading module: /conf/apps/dahua_mqtt.py:
2020-02-13 16:59:26.718658 WARNING AppDaemon: Removing associated apps:
2020-02-13 16:59:26.719152 WARNING AppDaemon: Dahua
2020-02-13 16:59:26.720614 INFO AppDaemon: App initialization complete

But not able to run the application i have in the config/apps,
from error.log:

2020-02-13 16:59:26.717624 WARNING Error: ------------------------------------------------------------

ModuleNotFoundError: No module named 'pycurl'
    import pycurl
  File "/conf/apps/dahua_mqtt.py", line 51, in 
  File "", line 219, in _call_with_frames_removed
  File "", line 783, in exec_module
  File "", line 671, in _load_unlocked
  File "", line 975, in _find_and_load_unlocked
  File "", line 991, in _find_and_load
  File "", line 1014, in _gcd_import
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    response = future.result()
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
2020-02-13 16:59:26.716987 WARNING Error: Traceback (most recent call last):
2020-02-13 16:59:26.714239 WARNING Error: ------------------------------------------------------------
2020-02-13 16:59:26.713758 WARNING Error: Unexpected error loading module: /conf/apps/dahua_mqtt.py:
2020-02-13 16:59:26.713064 WARNING Error: ------------------------------------------------------------
2020-02-13 16:44:05.105237 WARNING Error: ------------------------------------------------------------

Something with this pycurl?

This coming from dahua_mqtt.py
which was working perfectly on AD3:

github.com

algirdasc/appdaemon-apps/blob/e6d4a6364092d7b433cd49a7f338a5ca69f5566c/dahua_mqtt.py

"""
Dahua IP Camera events to MQTT app. Implemented from: https://github.com/johnnyletrois/dahua-watch

Example configuration:

DahuaMQTT:
  class: DahuaMQTT
  module: dahua_mqtt
  cameras:
    - host: 192.168.0.1
      port: 80
      user: user
      pass: pass
      topic: cameras/1
      events: VideoMotion,VideoBlind,VideoLoss,AlarmLocal,....
    - host: 192.168.0.2
      port: 80
      user: user
      pass: pass
      topic: cameras/2

This file has been truncated. show original

Burningstone · February 13, 2020, 3:14pm

Yes, pycurl is the issue. It’s not included in AppDaemon, so you need to install it separately.

Create a file requirements.txt which contains only:

pycurl

Add this file to the same directory as the appdaemon.yaml file, restart appdaemon and see if it works.

Avi_Cohen · February 13, 2020, 3:31pm

I thought it might be this one - I added it inside the AD container itself:
/usr/src/app/requirements.txt

Also create the requirements.txt as u suggest added the pycurl
with no change:

2020-02-13 17:35:18.990230 WARNING Error: ------------------------------------------------------------
2020-02-13 17:35:19.015446 WARNING Error: Unexpected error loading module: /conf/apps/dahua_mqtt.py:
2020-02-13 17:35:19.016081 WARNING Error: ------------------------------------------------------------
2020-02-13 17:35:19.019771 WARNING Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/conf/apps/dahua_mqtt.py", line 51, in <module>
    import pycurl
ModuleNotFoundError: No module named 'pycurl'

Please note ,
Under the apps folder
I have another requirements.txt (which was originally in AD 3) with the content of:

{
  "log_level": "info",
  "system_packages": [
    "libcurl",
    "python-dev"
  ],
  "python_packages": [
    "pip==20.0.2",
    "pycurl"
  ]
}

Maybe this requirements.txt should be format differently in AD 4?

Burningstone · February 13, 2020, 3:40pm

I took a look at the Dahua IP Cam topic and it seems that the requirements.txt file needs to look like this to work:

{
  "log_level": "info",
  "system_packages": [
    "libcurl",
    "python3-dev",
    "curl-dev",
    "gcc",
    "g++"
  ],
  "python_packages": [
    "pip==19.0.2",
    "pycurl"
  ]
}

Can you please try this? I’m also not sure whether it should be in the “apps” directory or in the same directory as “appdaemon.yaml”, try in both locations.

Avi_Cohen · February 13, 2020, 4:40pm

is there a different requirement for requirements.txt under config/apps
or under config?(right beside the appdaeomond.yaml)
or it actually doesn’t matter?

anyway , i copy the configuration as u suggest in both place ,
unfortunately no change:

2020-02-13 18:47:01.170525 WARNING Error: ------------------------------------------------------------
2020-02-13 18:47:01.169725 WARNING Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 1014, in _gcd_import
  File "", line 991, in _find_and_load
  File "", line 975, in _find_and_load_unlocked
  File "", line 671, in _load_unlocked
  File "", line 783, in exec_module
  File "", line 219, in _call_with_frames_removed
  File "/conf/apps/dahua_mqtt.py", line 51, in 
    import pycurl
ModuleNotFoundError: No module named 'pycurl'

Avi_Cohen · February 13, 2020, 5:13pm

@Burningstone
Thanks buddy !

only was i manage to overcome this annoying thing
is by install on appdaemond docker:
so if anyone else will face same issue ,
You need to enter the AD docker and do the installemet manually:

#get docker image by docker ls

docker exec -it 900b0167f8e9 sh
apk add curl-dev
pip3 install pycurl

I guess something with requiments.txt is not correct or missing…
Hopefully some1 can suggest better solution.

Avi_Cohen · February 25, 2020, 10:21am

One more issue found ,
It required also to config the namespace in appdaemon.yaml:

plugins:
    HASS:
      type: hass
      namespace: appdaemon-hass
      token: !secret appdaemon_token
      ha_url: "http://192.168.1.113:8123"
    MQTT:
      type: mqtt
      namespace: default
      client_host: !secret mqtt_broker

It is working now ,
but honestly got little tired of the AD,
I see 2 issues - the requirment.txt which suppose to simplify
the process - doesn’t work well .
(it doesn’t matter if the pycurl need more libraries or not- it just doesn’t work well)
Second ,what those namespace which added in AD v4?
I would expect it work with mqtt with minimum configuration … or at least see
it not able to publish to mqtt .

I wonder if there’s other option to work with dahua_mqtt.py without the appdemond …
hope some1 can make it work more natively.

Burningstone · February 25, 2020, 4:20pm

@eifinger can you help out here, I think you were the one that implemented it into AppDaemon right?

Check the docs here.

eifinger · February 25, 2020, 5:04pm

@Avi_Cohen it looks like you have a requirements.txt which holds a config for a Homeassistant (former hass.io) Addon.
The content of your requirements.txt should just be

pycurl

Without any JSON markup or system_packages and so on.

Burningstone · February 25, 2020, 5:16pm

Thanks for the insights.
And the file should be placed in the same directory as appdaemon.yaml right?

eifinger · February 25, 2020, 5:16pm

No they should be placed in the apps directory or any subdirectory of it

Avi_Cohen · February 27, 2020, 7:01pm

Although I already solved by install the pycurl inside the AD container,
i remove it and tried u suggestion :
create requirements.txt inside the app folder with pycurl
no other requirements.txt.

boot the AD ended with error:

2020-02-27 20:57:22.504789 WARNING Error: ------------------------------------------------------------
2020-02-27 20:57:22.503758 WARNING Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 1014, in _gcd_import
  File "", line 991, in _find_and_load
  File "", line 975, in _find_and_load_unlocked
  File "", line 671, in _load_unlocked
  File "", line 783, in exec_module
  File "", line 219, in _call_with_frames_removed
  File "/conf/apps/dahua_mqtt.py", line 51, in 
    import pycurl
**ModuleNotFoundError: No module named 'pycurl'**

Burningstone · February 27, 2020, 8:35pm

Did you try stopping the docker-compose stack and start it again? I think requirements will only be installed when the container is created.

eifinger · February 27, 2020, 8:37pm

That is correct.
Maybe there is a way to improve that for the future.

Avi_Cohen · March 28, 2020, 8:42pm

@Burningstone @eifinger
hopefully some of AD developers reading this :

I was doing a clean HA installation and thought for updating this threads ,
nothing was run - all dockers were down ,
remove appdaemond docker + all appdaemond images ,
i place pycurl string in the
/Dockers/appdaemon4/config/apps/requirements.txt

start the docker-container - new image download and install:
log show error remain:


2020-03-28 23:30:56.588462 WARNING Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/conf/apps/dahua_mqtt.py", line 51, in <module>
    import pycurl
ModuleNotFoundError: No module named 'pycurl'

after installing the pycurl manually as explain few post above - it solved.

eifinger · March 28, 2020, 8:54pm

Could you please post the full log of the container starting at the first start up?

Avi_Cohen · March 28, 2020, 9:27pm

sure :



2020-03-28 23:22:04.210486 WARNING Error: ------------------------------------------------------------
2020-03-28 23:30:56.578461 WARNING Error: ------------------------------------------------------------
2020-03-28 23:30:56.581837 WARNING Error: Unexpected error loading module: /conf/apps/dahua_mqtt.py:
2020-03-28 23:30:56.583230 WARNING Error: ------------------------------------------------------------
2020-03-28 23:30:56.588462 WARNING Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/conf/apps/dahua_mqtt.py", line 51, in <module>
    import pycurl
ModuleNotFoundError: No module named 'pycurl'

2020-03-28 23:30:56.589305 WARNING Error: ------------------------------------------------------------

Avi_Cohen · April 14, 2020, 10:05am

issue remain on 4.0.3
As WA i did the below:

docker-compose:

  appdaemon:
    build: 
      context: ./appdaemon
      dockerfile: Dockerfile
    container_name: appdaemon
    ports:
      - 5050:5050
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ${PWD}/appdaemon/config:/conf
    depends_on:
      - homeassistant
    restart: always

Add build instruction in appdaemon/Dockerfile:

FROM acockburn/appdaemon:latest
RUN apk update \
  && apk add \
    curl-dev
RUN pip install pycurl