Issue upgrading Appdamond to latest version

trying to update appdaemon to latest version and getting tons of errors
and appdaemon not started.
I saved previous appdaemon image so when i change the image to old one
so when i downgrade back - all is working.
I’d like to update appdaemon to latest version but not sure
what to change in my configuration :

docker-compose:

appdaemon:
     container_name: appdaemon
     image: avico78/appdaemon
     environment:
       - HA_URL="http://192.168.1.112:8123"
       - token="welcome"
     ports:
       - 5050:5050
     networks:
     - traefik_proxy  #as reverse proxy
     volumes:
       - ${PWD}/appdaemon/config:/conf
     depends_on:
       - homeassistant 

in app/requirements.txt

{
  "log_level": "info",
  "system_packages": [
    "libcurl",
    "python-dev"
  ],
  "python_packages": [
    "pip==20.0.2",
    "pycurl"
  ]
}

appdaemon.yaml

log:
  logfile: /conf/appdaemon.log
  errorfile: /conf/aderror.log
  accessfile: /conf/access.log
appdaemon:
  threads: 10
  app_dir: /conf/apps
  plugins:
    HASS:
      type: hass
      ha_url: "http://192.168.1.112:8123"
      token: '123123123.123123.234234-23424'
hadashboard:
  dash_url: http://192.168.1.113:5050
  dash_dir: /conf/dashboards

You need to correct your appdaemon.yaml, the configuration change quite a bit from 3 to 4. Check out my settings in another topic where I post a working appdaemon.yaml + docker-compose

1 Like

Thanks @Burningstone ,

I update the docker-compose as following:

  appdaemon_4:
      container_name: appdaemon_4
      image: acockburn/appdaemon:latest
      ports:
        - 5050:5050
      networks:
        - traefik_proxy
      volumes:
        - /etc/localtime:/etc/localtime:ro
        - ${PWD}/appdaemon4/config:/conf
      depends_on:
        - homeassistant
      restart: unless-stopped

In appdaemon.yaml

secrets: /conf/secrets.yaml
logs:
  main_log:
    filename: /conf/logs/appdaemon.log
  error_log:
    filename: /conf/logs/error.log
  access_log:
    filename: /conf/logs/access.log
appdaemon:
  time_zone: !secret timezone
  latitude: !secret latitude
  longitude: !secret longitude
  elevation: !secret elevation
  missing_app_warnings: 1
  production_mode: False
  app_dir: /conf/apps
  plugins:
    HASS:
      type: hass
      namespace: hass
      token: !secret appdaemon_token
      ha_url: "http://192.168.1.112:8123"
    MQTT:
      type: mqtt
      namespace: mqtt
      client_id: AD_4
      client_host: !secret mqtt_broker
http:
  url: http://192.168.1.112:5050
api:
hadashboard:
admin:
  title: AppDaemon
  stats_update: realtime

Appdaemon is up and running:

2020-02-13 16:59:24.390235 INFO HASS: Connected to Home Assistant 0.104.3
2020-02-13 16:59:25.506390 INFO HASS: Evaluating startup conditions
2020-02-13 16:59:25.619084 INFO AppDaemon: Got initial state from namespace hass
2020-02-13 16:59:26.685269 INFO AppDaemon: Scheduler running in realtime
2020-02-13 16:59:26.693511 INFO AppDaemon: Adding /conf/apps to module import path
2020-02-13 16:59:26.701070 INFO AppDaemon: Loading App Module: /conf/apps/dahua_mqtt.py
2020-02-13 16:59:26.718158 WARNING AppDaemon: Unexpected error loading module: /conf/apps/dahua_mqtt.py:
2020-02-13 16:59:26.718658 WARNING AppDaemon: Removing associated apps:
2020-02-13 16:59:26.719152 WARNING AppDaemon: Dahua
2020-02-13 16:59:26.720614 INFO AppDaemon: App initialization complete

But not able to run the application i have in the config/apps,
from error.log:

2020-02-13 16:59:26.717624 WARNING Error: ------------------------------------------------------------

ModuleNotFoundError: No module named 'pycurl'
    import pycurl
  File "/conf/apps/dahua_mqtt.py", line 51, in 
  File "", line 219, in _call_with_frames_removed
  File "", line 783, in exec_module
  File "", line 671, in _load_unlocked
  File "", line 975, in _find_and_load_unlocked
  File "", line 991, in _find_and_load
  File "", line 1014, in _gcd_import
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    response = future.result()
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
2020-02-13 16:59:26.716987 WARNING Error: Traceback (most recent call last):
2020-02-13 16:59:26.714239 WARNING Error: ------------------------------------------------------------
2020-02-13 16:59:26.713758 WARNING Error: Unexpected error loading module: /conf/apps/dahua_mqtt.py:
2020-02-13 16:59:26.713064 WARNING Error: ------------------------------------------------------------
2020-02-13 16:44:05.105237 WARNING Error: ------------------------------------------------------------

Something with this pycurl?

This coming from dahua_mqtt.py
which was working perfectly on AD3:

Yes, pycurl is the issue. It’s not included in AppDaemon, so you need to install it separately.

Create a file requirements.txt which contains only:

pycurl

Add this file to the same directory as the appdaemon.yaml file, restart appdaemon and see if it works.

I thought it might be this one - I added it inside the AD container itself:
/usr/src/app/requirements.txt

Also create the requirements.txt as u suggest added the pycurl
with no change:

2020-02-13 17:35:18.990230 WARNING Error: ------------------------------------------------------------
2020-02-13 17:35:19.015446 WARNING Error: Unexpected error loading module: /conf/apps/dahua_mqtt.py:
2020-02-13 17:35:19.016081 WARNING Error: ------------------------------------------------------------
2020-02-13 17:35:19.019771 WARNING Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/conf/apps/dahua_mqtt.py", line 51, in <module>
    import pycurl
ModuleNotFoundError: No module named 'pycurl'

Please note ,
Under the apps folder
I have another requirements.txt (which was originally in AD 3) with the content of:

{
  "log_level": "info",
  "system_packages": [
    "libcurl",
    "python-dev"
  ],
  "python_packages": [
    "pip==20.0.2",
    "pycurl"
  ]
}

Maybe this requirements.txt should be format differently in AD 4?

I took a look at the Dahua IP Cam topic and it seems that the requirements.txt file needs to look like this to work:

{
  "log_level": "info",
  "system_packages": [
    "libcurl",
    "python3-dev",
    "curl-dev",
    "gcc",
    "g++"
  ],
  "python_packages": [
    "pip==19.0.2",
    "pycurl"
  ]
}

Can you please try this? I’m also not sure whether it should be in the “apps” directory or in the same directory as “appdaemon.yaml”, try in both locations.

is there a different requirement for requirements.txt under config/apps
or under config?(right beside the appdaeomond.yaml)
or it actually doesn’t matter?

anyway , i copy the configuration as u suggest in both place ,
unfortunately no change:

2020-02-13 18:47:01.170525 WARNING Error: ------------------------------------------------------------
2020-02-13 18:47:01.169725 WARNING Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 1014, in _gcd_import
  File "", line 991, in _find_and_load
  File "", line 975, in _find_and_load_unlocked
  File "", line 671, in _load_unlocked
  File "", line 783, in exec_module
  File "", line 219, in _call_with_frames_removed
  File "/conf/apps/dahua_mqtt.py", line 51, in 
    import pycurl
ModuleNotFoundError: No module named 'pycurl'

@Burningstone
Thanks buddy !

only was i manage to overcome this annoying thing
is by install on appdaemond docker:
so if anyone else will face same issue ,
You need to enter the AD docker and do the installemet manually:

#get docker image by docker ls

docker exec -it 900b0167f8e9 sh
apk add curl-dev
pip3 install pycurl

I guess something with requiments.txt is not correct or missing…
Hopefully some1 can suggest better solution.

One more issue found ,
It required also to config the namespace in appdaemon.yaml:

plugins:
    HASS:
      type: hass
      namespace: appdaemon-hass
      token: !secret appdaemon_token
      ha_url: "http://192.168.1.113:8123"
    MQTT:
      type: mqtt
      namespace: default
      client_host: !secret mqtt_broker

It is working now ,
but honestly got little tired of the AD,
I see 2 issues - the requirment.txt which suppose to simplify
the process - doesn’t work well .
(it doesn’t matter if the pycurl need more libraries or not- it just doesn’t work well)
Second ,what those namespace which added in AD v4?
I would expect it work with mqtt with minimum configuration … or at least see
it not able to publish to mqtt .

I wonder if there’s other option to work with dahua_mqtt.py without the appdemond …
hope some1 can make it work more natively.

@eifinger can you help out here, I think you were the one that implemented it into AppDaemon right?

Check the docs here.

@Avi_Cohen it looks like you have a requirements.txt which holds a config for a Homeassistant (former hass.io) Addon.
The content of your requirements.txt should just be

pycurl

Without any JSON markup or system_packages and so on.

Thanks for the insights.
And the file should be placed in the same directory as appdaemon.yaml right?

No they should be placed in the apps directory or any subdirectory of it

Although I already solved by install the pycurl inside the AD container,
i remove it and tried u suggestion :
create requirements.txt inside the app folder with pycurl
no other requirements.txt.

boot the AD ended with error:

2020-02-27 20:57:22.504789 WARNING Error: ------------------------------------------------------------
2020-02-27 20:57:22.503758 WARNING Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 1014, in _gcd_import
  File "", line 991, in _find_and_load
  File "", line 975, in _find_and_load_unlocked
  File "", line 671, in _load_unlocked
  File "", line 783, in exec_module
  File "", line 219, in _call_with_frames_removed
  File "/conf/apps/dahua_mqtt.py", line 51, in 
    import pycurl
**ModuleNotFoundError: No module named 'pycurl'**

Did you try stopping the docker-compose stack and start it again? I think requirements will only be installed when the container is created.

That is correct.
Maybe there is a way to improve that for the future.

@Burningstone @eifinger
hopefully some of AD developers reading this :

I was doing a clean HA installation and thought for updating this threads ,
nothing was run - all dockers were down ,
remove appdaemond docker + all appdaemond images ,
i place pycurl string in the
/Dockers/appdaemon4/config/apps/requirements.txt

start the docker-container - new image download and install:
log show error remain:


2020-03-28 23:30:56.588462 WARNING Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/conf/apps/dahua_mqtt.py", line 51, in <module>
    import pycurl
ModuleNotFoundError: No module named 'pycurl'

after installing the pycurl manually as explain few post above - it solved.

Could you please post the full log of the container starting at the first start up?

sure :



2020-03-28 23:22:04.210486 WARNING Error: ------------------------------------------------------------
2020-03-28 23:30:56.578461 WARNING Error: ------------------------------------------------------------
2020-03-28 23:30:56.581837 WARNING Error: Unexpected error loading module: /conf/apps/dahua_mqtt.py:
2020-03-28 23:30:56.583230 WARNING Error: ------------------------------------------------------------
2020-03-28 23:30:56.588462 WARNING Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 788, in check_app_updates
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/local/lib/python3.8/site-packages/appdaemon/utils.py", line 276, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/appdaemon/app_management.py", line 580, in read_app
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/conf/apps/dahua_mqtt.py", line 51, in <module>
    import pycurl
ModuleNotFoundError: No module named 'pycurl'

2020-03-28 23:30:56.589305 WARNING Error: ------------------------------------------------------------

issue remain on 4.0.3
As WA i did the below:

docker-compose:

  appdaemon:
    build: 
      context: ./appdaemon
      dockerfile: Dockerfile
    container_name: appdaemon
    ports:
      - 5050:5050
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ${PWD}/appdaemon/config:/conf
    depends_on:
      - homeassistant
    restart: always

Add build instruction in appdaemon/Dockerfile:

FROM acockburn/appdaemon:latest
RUN apk update \
  && apk add \
    curl-dev
RUN pip install pycurl