DSMR Reader Add-on for Home Assistant

sanderdw · January 4, 2022, 6:44pm

Yes i had to push the 0.7.1 update as it was not working anymore for fresh installs. Which error do you get?

Alfagek · January 4, 2022, 6:57pm

@sanderdw , see also your github https://github.com/sanderdw/hassio-addons/issues/26#issuecomment-1005070611


Starting DSMR Reader - datalogger... 127.0.0.1 - - [04/Jan/2022:18:06:30 +0100] "GET /about HTTP/1.1" 200 14949 "-" "curl/7.79.1" "-" Current logging level set to "ERROR". More information can be found here: https://dsmr-reader.readthedocs.io/en/latest/how-to/troubleshooting/enabling-debug-logging.html 2022-01-04 18:06:31,345 ERROR mixins run_once 103 | dsmr_datalogger.management.commands.dsmr_datalogger: [!] Exception raised. Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/serial/serialposix.py", line 322, in open self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) PermissionError: [Errno 13] Permission denied: '/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK43ODEX-if00-port0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/dsmr_datalogger/scripts/dsmr_datalogger_api_client.py", line 34, in read_telegram serial_handle = serial.serial_for_url(url=url_or_port, timeout=MAX_READ_TIMEOUT, **serial_kwargs) File "/usr/local/lib/python3.10/site-packages/serial/__init__.py", line 90, in serial_for_url instance.open() File "/usr/local/lib/python3.10/site-packages/serial/serialposix.py", line 325, in open raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg)) serial.serialutil.SerialException: [Errno 13] could not open port /dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK43ODEX-if00-port0: [Errno 13] Permission denied: '/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK43ODEX-if00-port0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/dsmr_backend/mixins.py", line 92, in run_once self.run(data=self.data, **options) File "/app/dsmr_datalogger/management/commands/dsmr_datalogger.py", line 29, in run telegram = next(self.telegram_generator) File "/app/dsmr_datalogger/scripts/dsmr_datalogger_api_client.py", line 36, in read_telegram raise RuntimeError('Failed to connect: {}', error) RuntimeError: ('Failed to connect: {}', SerialException(13, "could not open port /dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK43ODEX-if00-port0: [Errno 13] Permission denied: '/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AK43ODEX-if00-port0'"))

sanderdw · January 4, 2022, 8:44pm

Just published 0.7.2, let me know if it works.

ygeffens · January 4, 2022, 9:13pm

Hi Sander

I’m having the same problem (autoupdated to 0.7.1)
How can I initiate a manual update?
HA says there is no update available.

Thanks
Yves

sanderdw · January 4, 2022, 9:31pm

In the “Add-on Store” you have to click on the 3 dots in the top right and choose Reload/Herlaad.

ygeffens · January 4, 2022, 9:48pm

Thanks, didn’t know. It’s working again.
And I disabled auto-update

I also replaced
/dev/ttyUSB0
with
/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AQ4P7FZR-if00-port0

I already did that when it wasn’t working on 0.7.1 (in an attempt to solve the problem), but I leave it like this. I think it’s safer (this should not change, while /dev/ttyUSB0 can).

Alfagek · January 5, 2022, 2:48am

@sanderdw thanks for the quick fix
Running on both systems

fxx · January 9, 2022, 3:48pm

After updating to 0.7.2, my DMSR Reader add-on keeps auto-stopping after a minute or so.

Thiings I have tried so far:
-reinstall DSMR Reader add-on
-stop/start PostgressDB

After restarting both add-ons I am able to briefly get into the WebUI.
The Log throws the following:

`2022-01-09 16:38:43,016 ERROR    mixins       run_once                        103 | dsmr_datalogger.management.commands.dsmr_datalogger: [!] Exception raised. Traceback (most recent call last):
  File "/dsmr/dsmr_backend/mixins.py", line 92, in run_once
    self.run(data=self.data, **options)
  File "/dsmr/dsmr_datalogger/management/commands/dsmr_datalogger.py", line 29, in run
    telegram = next(self.telegram_generator)
  File "/dsmr/dsmr_datalogger/scripts/dsmr_datalogger_api_client.py", line 44, in read_telegram
    raise RuntimeError(
RuntimeError: It took too long to detect a telegram. Check connection params. Bytes currently in buffer: 1618
2022-01-09 16:38:48,767 INFO exited: dsmr_datalogger (exit status 0; expected)
2022-01-09 16:38:48,767 INFO exited: dsmr_datalogger (exit status 0; expected)
2022-01-09 16:38:49,776 INFO spawned: 'dsmr_datalogger' with pid 47
2022-01-09 16:38:49,776 INFO spawned: 'dsmr_datalogger' with pid 47
2022-01-09 16:38:49,778 WARN received SIGQUIT indicating exit request
2022-01-09 16:38:49,778 WARN received SIGQUIT indicating exit request
2022-01-09 16:38:49,780 INFO waiting for processes, dsmr_backend, dsmr_datalogger, dsmr_webinterface, nginx to die
2022-01-09 16:38:49,780 INFO waiting for processes, dsmr_backend, dsmr_datalogger, dsmr_webinterface, nginx to die
2022-01-09 16:38:49,807 INFO stopped: nginx (exit status 0)
2022-01-09 16:38:49,807 INFO stopped: nginx (exit status 0)
[2022-01-09 16:38:49 +0100] [40] [INFO] Handling signal: term
[2022-01-09 16:38:49 +0100] [46] [INFO] Worker exiting (pid: 46)
[2022-01-09 16:38:50 +0100] [40] [INFO] Shutting down: Master
2022-01-09 16:38:50,496 INFO stopped: dsmr_webinterface (exit status 0)
2022-01-09 16:38:50,496 INFO stopped: dsmr_webinterface (exit status 0)
2022-01-09 16:38:51,498 INFO success: dsmr_datalogger entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-01-09 16:38:51,498 INFO success: dsmr_datalogger entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-01-09 16:38:51,506 INFO stopped: dsmr_datalogger (terminated by SIGTERM)
2022-01-09 16:38:51,506 INFO stopped: dsmr_datalogger (terminated by SIGTERM)
2022-01-09 16:38:52,778 INFO stopped: dsmr_backend (exit status 0)
2022-01-09 16:38:52,778 INFO stopped: dsmr_backend (exit status 0)
2022-01-09 16:38:52,780 INFO stopped: processes (terminated by SIGTERM)
2022-01-09 16:38:52,780 INFO stopped: processes (terminated by SIGTERM)`

Is there something wrong with the database? I’m clearly missing something here

sanderdw · January 10, 2022, 10:20am

You need to provide some more information, which hardware, dsmr configuration etc? Maybe you need to double check the USB port?

fxx · January 11, 2022, 10:18am

Hi Sander, I use the Wemos P1 reader v3.1 over wifi (ESP link).

DSMR Reader config:


DSMRREADER_ADMIN_USER: admin
DSMRREADER_ADMIN_PASSWORD: 
DJANGO_DATABASE_NAME: dsmrreader
DJANGO_DATABASE_USER: postgres
DJANGO_DATABASE_PASSWORD:
DJANGO_DATABASE_HOST: 77b2833f-timescaledb
DJANGO_DATABASE_PORT: '5432'
DATALOGGER_MODE: standalone
DATALOGGER_SERIAL_PORT: /dev/ttyUSB1
DATALOGGER_INPUT_METHOD: ipv4
DATALOGGER_SERIAL_BAUDRATE: '115200'
DATALOGGER_NETWORK_HOST: 192.168.0.132
DATALOGGER_NETWORK_PORT: '23'

The energy dashboard in HA keeps updating, which indicates that the data flow grom the P1 reader is stil ok.

TimescaleDB config:


databases:
  - homeassistant
  - dsmrreader
timescale_enabled:
  - homeassistant
timescaledb:
  telemetry: basic
  maxmemory: 512MB
  maxcpus: '4'
max_connections: 20
system_packages: []
init_commands: []

rednas · January 11, 2022, 10:31am

Hello,

I am having the same kind of problems I guess. Every minute or so the plugin restarts. I can see this in the logs.
This is the last log I could copy:

[ INFO ] DSMR release: v4.19.0

[ INFO ] Removing existing PID files...

[ INFO ] Creating log directory...

[ INFO ] Setting architecture requirements...

[ INFO ] X64 Architecture

[ INFO ] Enabling IFrame...

[ INFO ] Fixing /dev/ttyUSB* security...
chmod: /dev/ttyUSB0: Read-only file system

[ INFO ] Verifying if the DSMR web credential variables have been set...

[ INFO ] Verifying database connectivity to host: 77b2833f-timescaledb with port: 5432...

[ INFO ] Database connectivity successfully verified!

[ INFO ] Running post configuration...
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, dsmr_api, dsmr_backend, dsmr_backup, dsmr_consumption, dsmr_datalogger, dsmr_dropbox, dsmr_frontend, dsmr_influxdb, dsmr_mindergas, dsmr_mqtt, dsmr_notification, dsmr_pvoutput, dsmr_stats, dsmr_weather, sessions
Running migrations:
  No migrations to apply.
563 static files copied to '/var/www/dsmrreader/static'.
Updating password of superuser "XXXXXX"
Deactivating any other existing superusers

[ INFO ] Checking for NGINX SSL configuration...

[ INFO ] ENABLE_NGINX_SSL is disabled, nothing to see here. Continuing...

[ INFO ] Checking for HTTP AUTHENTICATION configuration...

[ INFO ] ENABLE_HTTP_AUTH is disabled, nothing to see here. Continuing...

[ INFO ] Configuring DSMR in standlone datalogger mode....

[ INFO ] Starting supervisord...

[ INFO ] Logfiles can be found at: /var/log/supervisor/*.log and /tmp/supervisord.log
2022-01-11 11:22:02,946 INFO Set uid to user 0 succeeded
2022-01-11 11:22:02,946 INFO Set uid to user 0 succeeded
2022-01-11 11:22:02,958 INFO RPC interface 'supervisor' initialized
2022-01-11 11:22:02,958 INFO RPC interface 'supervisor' initialized
2022-01-11 11:22:02,959 INFO supervisord started with pid 35
2022-01-11 11:22:02,959 INFO supervisord started with pid 35
2022-01-11 11:22:03,964 INFO spawned: 'processes' with pid 37
2022-01-11 11:22:03,964 INFO spawned: 'processes' with pid 37
2022-01-11 11:22:03,969 INFO spawned: 'dsmr_backend' with pid 38
2022-01-11 11:22:03,969 INFO spawned: 'dsmr_backend' with pid 38
2022-01-11 11:22:03,976 INFO spawned: 'dsmr_datalogger' with pid 39
2022-01-11 11:22:03,976 INFO spawned: 'dsmr_datalogger' with pid 39
2022-01-11 11:22:03,984 INFO spawned: 'dsmr_webinterface' with pid 40
2022-01-11 11:22:03,984 INFO spawned: 'dsmr_webinterface' with pid 40
2022-01-11 11:22:03,991 INFO spawned: 'nginx' with pid 41
2022-01-11 11:22:03,991 INFO spawned: 'nginx' with pid 41
[2022-01-11 11:22:04 +0100] [40] [INFO] Starting gunicorn 20.1.0
[2022-01-11 11:22:04 +0100] [40] [INFO] Listening at: unix:/var/tmp/gunicorn--dsmr_webinterface.socket (40)
[2022-01-11 11:22:04 +0100] [40] [INFO] Using worker: sync
[2022-01-11 11:22:04 +0100] [46] [INFO] Booting worker with pid: 46
2022-01-11 11:22:05,611 INFO success: processes entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-01-11 11:22:05,611 INFO success: processes entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-01-11 11:22:05,613 INFO success: dsmr_backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-01-11 11:22:05,613 INFO success: dsmr_backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-01-11 11:22:05,615 INFO success: dsmr_datalogger entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-01-11 11:22:05,615 INFO success: dsmr_datalogger entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Current logging level set to "ERROR". More information can be found here: https://dsmr-reader.readthedocs.io/en/latest/how-to/troubleshooting/enabling-debug-logging.html
Current logging level set to "ERROR". More information can be found here: https://dsmr-reader.readthedocs.io/en/latest/how-to/troubleshooting/enabling-debug-logging.html
2022-01-11 11:22:07,140 INFO success: dsmr_webinterface entered RUNNING state, process has stayed up for > than 3 seconds (startsecs)
2022-01-11 11:22:07,140 INFO success: dsmr_webinterface entered RUNNING state, process has stayed up for > than 3 seconds (startsecs)
2022-01-11 11:22:09,143 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2022-01-11 11:22:09,143 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2022-01-11 11:26:24,611 ERROR    mixins       run_once                        103 | dsmr_datalogger.management.commands.dsmr_datalogger: [!] Exception raised. Traceback (most recent call last):
  File "/dsmr/dsmr_backend/mixins.py", line 92, in run_once
    self.run(data=self.data, **options)
  File "/dsmr/dsmr_datalogger/management/commands/dsmr_datalogger.py", line 29, in run
    telegram = next(self.telegram_generator)
  File "/dsmr/dsmr_datalogger/scripts/dsmr_datalogger_api_client.py", line 50, in read_telegram
    incoming_bytes = serial_handle.read(MAX_BYTES_PER_READ)
  File "/usr/local/lib/python3.10/site-packages/serial/serialposix.py", line 595, in read
    raise SerialException(
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?)
2022-01-11 11:26:29,855 INFO exited: dsmr_datalogger (exit status 0; expected)
2022-01-11 11:26:29,855 INFO exited: dsmr_datalogger (exit status 0; expected)
2022-01-11 11:26:30,860 INFO spawned: 'dsmr_datalogger' with pid 47
2022-01-11 11:26:30,860 INFO spawned: 'dsmr_datalogger' with pid 47
2022-01-11 11:26:30,862 WARN received SIGQUIT indicating exit request
2022-01-11 11:26:30,862 WARN received SIGQUIT indicating exit request
2022-01-11 11:26:30,863 INFO waiting for processes, dsmr_backend, dsmr_datalogger, dsmr_webinterface, nginx to die
2022-01-11 11:26:30,863 INFO waiting for processes, dsmr_backend, dsmr_datalogger, dsmr_webinterface, nginx to die
2022-01-11 11:26:30,880 INFO stopped: nginx (exit status 0)
2022-01-11 11:26:30,880 INFO stopped: nginx (exit status 0)
[2022-01-11 11:26:30 +0100] [40] [INFO] Handling signal: term
[2022-01-11 11:26:30 +0100] [46] [INFO] Worker exiting (pid: 46)
[2022-01-11 11:26:31 +0100] [40] [INFO] Shutting down: Master
2022-01-11 11:26:32,253 INFO success: dsmr_datalogger entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-01-11 11:26:32,253 INFO success: dsmr_datalogger entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-01-11 11:26:32,255 INFO stopped: dsmr_webinterface (exit status 0)
2022-01-11 11:26:32,255 INFO stopped: dsmr_webinterface (exit status 0)
Current logging level set to "ERROR". More information can be found here: https://dsmr-reader.readthedocs.io/en/latest/how-to/troubleshooting/enabling-debug-logging.html2022-01-11 11:26:32,379 INFO stopped: dsmr_datalogger (terminated by SIGTERM)
2022-01-11 11:26:32,379 INFO stopped: dsmr_datalogger (terminated by SIGTERM)
2022-01-11 11:26:33,626 INFO stopped: dsmr_backend (exit status 0)
2022-01-11 11:26:33,626 INFO stopped: dsmr_backend (exit status 0)
2022-01-11 11:26:33,627 INFO stopped: processes (terminated by SIGTERM)
2022-01-11 11:26:33,627 INFO stopped: processes (terminated by SIGTERM)

This is my TimescaleDB config:

databases:
  - homeassistant
  - dsmrreader
timescale_enabled:
  - homeassistant
  - dsmrreader
timescaledb:
  telemetry: basic
  maxmemory: 512MB
  maxcpus: '4'
max_connections: 20
system_packages: []
init_commands: []

(A side question: does the dsmrreader database need timescale enabled?)

Does anyone know what is going on and how I can fix this??

sanderdw · January 11, 2022, 10:46am

No don’t put the db in timescale_enabled, it creates only overhead which you don’t need. Which exact hardware are you guys running?

rednas · January 11, 2022, 11:45am

Ok, thanks I removed it from the timescale_enabled as I’m not running it for long and don’t mind that my data is probably gone.

I am running HA OS 7.1 with Core 2021.12.8 on an Intel NUC with Proxmox. Do you require more information?

fxx · January 11, 2022, 9:26pm

I run HAOS 2021.12.9 on a Dell Wase 5020 with 32gb flashdrive which according to HA is about 61% full.a

rednas · January 12, 2022, 1:27pm

Maybe my problems are because I am also using the port for

- platform: dsmr

I also tried adding my MQTT settings to the DSMR reader to check whether it works in Home Assistant, but I entered something wrong. I can see in the logs that DSMR reader is again restarting a lot, but now I am unable to enter DSMR reader completely. Sometimes the login page shows up, but when I enter it probably restarts already and it shows that the page is unreachable… (I also tried removing and readding it, but it looks like it remembers the MQTT settings)
So basically I am locked out now… How do I change my MQTT settings now?

sanderdw · January 12, 2022, 2:32pm

You can’t run both this and the DSMR Reader addon as they both need the USB port. With this integration you can get the data back in HA: DSMR Reader - Home Assistant

You can try to change the MQTT settings in the database, it’s in the dsmr_mqtt_mqttbrokersettings table.

rednas · January 12, 2022, 3:09pm

Thanks. I think I got it to work.
However I don’t get the data back in HA with the link you provided. All sensors are created, but all of them show “unknown”.
MQTT log shows:

1641999673: New client connected from 172.30.32.1 as DSMR-reader (p2, c1, k60, u'mosquitto').
1641999764: Client DSMR-reader has exceeded timeout, disconnecting.

Does this matter? Do you know what I could be doing wrong?

ADDED LATER:
I also found this in the DSMR Reader settings, so there are 0 outgoing MQTT messages. Why is this? I selected all of the split topics under MQTT.

soepkip · January 14, 2022, 1:02pm

This looks interesting! Thanks for developing & maintaining this!

I am currently running a ‘native’ DSMR reader on a Rpi3.
I am planning to install HA on a Rpi3 (or 4) and integrate DSMR Reader into that…

Would I be able to restore my DSMR reader database into this HA-installed Add-on version?
(Because it seems like a Docker installed ‘in’ HA, a bit tricky maybe…)

So something like:

Get a fresh Pi, install Raspbian & update etc.
Install HA
Install DSMRR-Add On
Restore dsmr DB (how?)

nelbs · January 14, 2022, 5:59pm

Yes that’s possible. I did it through the console of the timescale DB container. See this issue

github.com/xirixiz/dsmr-reader-docker

restore backup with hassio dsmr-addon

opened 09:10PM - 04 Mar 21 UTC

closed 12:17PM - 11 Mar 21 UTC

nelbs

bug

Ik heb onlangs de Homeassistant [addon dsmr-reader ](https://github.com/sanderdw…/hassio-addons)geinstalleerd. Alles werkt naar behoren maar het herstellen van een backup van mijn oude installatie lukt me niet. Ik weet dat ik niet direct deze container gebruik maar wellicht dat iemand mij verder kan helpen. ## Setup/Architecture information Pi4 Homeassistant supervised met DSMR-reader en TimescaleDB addons ## Version of the Docker image Addon dsmr reader: 0.1.2 addon timescale DB 1.1.6 ## Configuration nvt ## Describe the bug Ik heb het volgende allemaal geprobeerd: **1)** Als ik de container probeer te stoppen met docker compose stop krijg ik de volgende melding: ``` ERROR: Can't find a suitable configuration file in this directory or any parent. Are you in the right directory? Supported filenames: docker-compose.yml, docker-compose.yaml ``` ergens is dit logisch aangezien ik de container niet met compose heb opgezet... **2)** als ik de container via portainer stop en vervolgens de backup wil terug zetten met het volgende commando: `docker exec -t addon_0493853b_dsmr_reader createdb -O dsmrreader dsmrreader -U dsmrreader` krijg ik de volgende melding: `Error response from daemon: Container ..... is not running` **3)** Als ik hetzelfde commando geef als de container draait dan krijg ik geen melding maar van een backup is ook geen sprake. ## Debug log ```text Add your logs here. ```

sanderdw · January 17, 2022, 2:42pm

2022-01-17: 0.7.3 Image update and debug option