Support for reading Dutch Smart Meter (electricity/gas) (P1 port)

Feature Requests
61

Both graphite is the de-facto standard in the industry for storing time series data so I wouldn’t call it obscure ;). It is a different database type then mysql because mysql is not optimal for storing this kind of data.

Graphite consist of 2 parts: carbon (data collection and storage) and graphite (web interface to display metrics). For carbon I would suggest installing: https://github.com/lomik/go-carbon which is a little bit more efficient then the default carbon and for graphite you can choose: http://graphite.readthedocs.io/en/latest/install.html or https://graphite-api.readthedocs.io/en/latest/installation.html

I have run a complete graphite + grafana setup on a RPi 1B before, it should be doable, although I don’t know if you could fit graphite + hass on one RPi memory wise.

62

Can you try installing this version: pip3 install https://github.com/aequitas/dsmr_parser/archive/error_handling.zip --target .homeassistant/deps/ --upgrade

It does not solve the parse problem, but it should prevent reading to cease after a parse failure.

Also I think the DSMR 4 includes a CRC at the end of the telegram, after the !. I have not searched in the spec if that is actually the case. But if it is maybe we could use it to determine if a telegram is valid before parsing it.

63

I changed the internals of the component a little bit to provide better integration with asyncio. Newest version can be found here to be installed in your custom_components directory: https://raw.githubusercontent.com/aequitas/home-assistant/dsmr/homeassistant/components/sensor/dsmr.py

@fversteegen: this change includes the error handling I worked on earlier.

64

@aequitas Hi Johan, would this component also work via a wifi enables P1 (http://romix.macuser.nl/ )

65

Good progress made here I notice! Thanks for the hard work! In the pull request I noticed it’s probably going to make 0.34. That’s great! :slight_smile:
I was wondering, does it also give us current usage sensors? If not, it shouldn’t be to hard to code that in, right?

Again, thanks for the hard work guys :thumbsup:

66

Thanks @aequitas! Trying the fix right now.

@ultrasub also the current electricity usage but only the total gas reading. Am working on breaking this down into the hourly usage.

Edit: the new version doesn’t appear to work as expected. See the log below:

16-11-21 20:06:09 dsmr_parser.protocol: failed to parse telegram
Traceback (most recent call last):
File “/home/hass/.homeassistant/deps/dsmr_parser/protocol.py”, line 82, in handle_lines
parsed_telegram = self.telegram_parser.parse(self.telegram)
File “/home/hass/.homeassistant/deps/dsmr_parser/parsers.py”, line 32, in parse
obis_reference, dsmr_object = self.parse_line(line_value.strip())
File “/home/hass/.homeassistant/deps/dsmr_parser/parsers.py”, line 47, in parse_line
return obis_reference, parser.parse(line_value)
File “/home/hass/.homeassistant/deps/dsmr_parser/parsers.py”, line 135, in parse
return CosemObject(self._parse(line))
File “/home/hass/.homeassistant/deps/dsmr_parser/parsers.py”, line 84, in _parse
raise ParseError(“Invalid ‘%s’ line for ‘%s’”, line, self)
dsmr_parser.exceptions.ParseError: (“Invalid ‘%s’ line for ‘%s’”, ‘\x7f05\x02RJ2r)S&\x170!J)Ha15\x02R\x1a\nr:r\r\x13%\x1802A%jHx15\x02R\x12\nri\x13\t&J009\x1a"RikUJC\n15\x02R\x12\x12ri\x13\t&J009\x02\x02\x02RikUJC\n05\nR\x12"r)\x13\t&J00MJ)Ha05\nRJ2r)\x13\t&J47M\x02\x1a\x02\x1a"\x1a\x1a\x02\x1a\x02\x1a\n\x1a2\x1a\x02\x1aB\x1a\n\x1a\x02\x1a:\x1a\x12\x1a\n\x1a2Jjlx05\nR\x12"rI\x13)&J16E\n\x12\n\x12\x02\x02\x02\x02\x02:\x15\t%L008ar\t\x13L&\x15mMJ)Ha!\x19\x15\x05"jlx/a5\x05U\x13j\x14(!FG10E\x12\x1a\x02\x12\x1aJjH(Ha1-3:0.2.8!"\x12J)Hx05\x02R\nr\t\x13\t&J16E\n\x12\n\x12\n\x02\x12*\x12:\x15)Ha05\x02RJ2r\nr)\x13J\x1a53A\x1a\x02\x1a\x1a\x1a\n\x1a\x02\x1a\x02\x1a\x1a\x1a\x12\x1a:\x1a:\x1aB\x1aJ\x1aB\x1aB\x1a\n\x1a2J)Ha15\x02R\nr\tr)\x13J\x180A\n\n"r\tB2Rik\x15ZJ’, <dsmr_parser.parsers.CosemParser object at 0xad5beed0>)
16-11-21 20:14:30 dsmr_parser.protocol: failed to parse telegram
Traceback (most recent call last):
File “/home/hass/.homeassistant/deps/dsmr_parser/protocol.py”, line 82, in handle_lines
parsed_telegram = self.telegram_parser.parse(self.telegram)
File “/home/hass/.homeassistant/deps/dsmr_parser/parsers.py”, line 32, in parse
obis_reference, dsmr_object = self.parse_line(line_value.strip())
File “/home/hass/.homeassistant/deps/dsmr_parser/parsers.py”, line 47, in parse_line
return obis_reference, parser.parse(line_value)
File “/home/hass/.homeassistant/deps/dsmr_parser/parsers.py”, line 135, in parse
return CosemObject(self._parse(line))
File “/home/hass/.homeassistant/deps/dsmr_parser/parsers.py”, line 84, in _parse
raise ParseError(“Invalid ‘%s’ line for ‘%s’”, line, self)
dsmr_parser.exceptions.ParseError: (“Invalid ‘%s’ line for ‘%s’”, ‘\x7f0-ARJ2r)\x13M\x170!\x02\x02\x02\nJjlx15\x02R\nri\x13\t&J00.M"*RikUJC\x0f15\x02R\x12ri\x13\t&J009\x02\x02\x02RZkUJC\x0f0-ARJ2ri\x13I&L(00A\x02\x12J)Ha0-0:96.7.9!\x02\x02\x02\x02\x02J)Ha15\x02RJJrJ:r\t\x13J\x18)!\x02j\t\x13’N6.]r)SNJ’, <dsmr_parser.parsers.CosemParser object at 0xad5f0270>)

67

Hurray I’ve got some decent dashboarding going on using InfluxDb and Grafana!

Influxdb was easy to install, just a small hiccup where they didn;t very abondantly mention that the web ui has been deprecated.

Grafana took some more effort, but was done using https://github.com/fg2it/grafana-on-raspberry

Now this has been set up, I eagerly await your pull request to be accepted.

68

Like @fversteegen mentioned, for power usage/production yes. For gas the Power meter only ‘queries’ the Gas meter every hour. So the numbers are not accurate enough to give nice graphs like the power meter does. It would be possible to add a derivative of the Gas meter reading to create a hourly usage as mentioned.

69

I didn’t expect the parse error to disappear, I thinks all the \x's are messing stuff up. Don’t know what they are, maybe invalidly encoded characters? Could be that sometimes the serial line gets misaligned. Which settings are you currently using?

Looking at the timestamp of the logging it appears the reading is now continuing after error, right?

70

Great job. Could you be so nice to try the component out with your setup, just install this file in your /custom_components folder: https://raw.githubusercontent.com/aequitas/home-assistant/dsmr/homeassistant/components/sensor/dsmr.py

I would like to know if it works for as many people before having it merged.

71

No @aequitas it did not continue any reading. I did try a cu -l and see the telegrams coming in fine. I also tried a full reboot but with similar results.

Just to check; did you change anything in the needs for the configuration.yaml? Mine reads the following under sensor:

  • name: dsmr
    platform: dsmr
    port: /dev/ttyUSB-P1
    dsmr_version: 4
72

Configuration looks fine. Do the settings that work for you with other program match these: https://github.com/ndokter/dsmr_parser/blob/master/dsmr_parser/serial.py#L23-L31 ?

Maybe it does not continue reading because it cannot assemble any valid telegram anymore. Does it at least keep printing those dsmr_parser.protocol: failed to parse telegram error lines every now and then (probably not every 10 seconds) without having to restart hass? In that case the error handling would work.

73

I decided to start from, more or less, scratch. I removed all dependencies, updated the raspberry, upgraded Home Assistant, checked if it all worked OK and downloaded the dsmr.py file and… All is working flawlessly now :slight_smile: I don’t know which component was causing the issues but I have been running like this for a few hours now without a stoppage of the signal and without any errors in the logs!

74

Good to hear. Maybe there was a a file stuck that never got updated. Let me know when you do experience problems again, then we will do a chat session on the HA gitter and should solve your problems in no time.

In the mean time the component has been accepted by HA maintainers and should be included in the next release.

75

I just installed the DMR branch I pulled from your repo, and while some of the reading works nice, I get a few of these parsing errors too:

ERROR:dsmr_parser.protocol:failed to parse telegram
Traceback (most recent call last):
  File "/home/pi/.homeassistant/deps/dsmr_parser/protocol.py", line 82, in handle_lines
    parsed_telegram = self.telegram_parser.parse(self.telegram)
  File "/home/pi/.homeassistant/deps/dsmr_parser/parsers.py", line 32, in parse
    obis_reference, dsmr_object = self.parse_line(line_value.strip())
  File "/home/pi/.homeassistant/deps/dsmr_parser/parsers.py", line 47, in parse_line
    return obis_reference, parser.parse(line_value)
  File "/home/pi/.homeassistant/deps/dsmr_parser/parsers.py", line 135, in parse
    return CosemObject(self._parse(line))
  File "/home/pi/.homeassistant/deps/dsmr_parser/parsers.py", line 84, in _parse
    raise ParseError("Invalid '%s' line for '%s'", line, self)
dsmr_parser.exceptions.ParseError: ("Invalid '%s' line for '%s'", '1.8.1(000757.633*kWh)\r:1.8.2(000866.884*1-0:2.8.1(000000.000*kWh)', <dsmr_parser.parsers.CosemParser object at 0xb31e8eb0>)

and:

dsmr_parser.exceptions.ParseError: ("Invalid '%s' line for '%s'", '1.0(MYMETERNUMBERHERE-1:24.2.1(161123230000W)(00598./KFM5KAIFA-METER', <dsmr_parser.parsers.MBusParser object at 0xb31ec790>)

and

dsmr_parser.exceptions.ParseError: ("Invalid '%s' line for '%s'", ':2.8.1(000000.000*kWh)1-0:2.8.2(0-0:96.14.0(0001)', <dsmr_parser.parsers.CosemParser object at 0xb31e8f10>)

I used config:

  - name: dsmr
    platform: dsmr
    port: /dev/ttyUSB0
    dsmr_version: 4

Could this be due to an invalidly selected firmware? How can I find out what firmware I have?

When I installed your dsmr console, I was able to read the data using both:

dsmr_console -v --device /dev/ttyUSB0 --version 4 -p N -b 8

but also with

dsmr_console -v --device /dev/ttyUSB0 --version 4

(which resulted in even parity and a bytesize of 7)

However after installing your update all is well, and no error messages are displayed!

76

@Atreyu which smart meter do you have? In the user manual it should describe the DSMR version and port settings.

I now ran several hours with the new DSMR script and everything looks fine. If I might already propose some feature requests (I know: beggars can’t be choosers) my wish list would be:

But already: hiphip hooray for the great work of @aequitas!

77

I’m a little confused, in the beginning of your message you say it has errors, But the last line says it works now. Can I conclude you are not having parsing issues atm?

Also the code got merged into the HA dev branch. So no need fetching it from my repository anymore :).

78

Klagen mag altijd :wink:

Good to hear it works well.

  • For the errors correction, could you open that as a feature on the dsmr_parser project (https://github.com/ndokter/dsmr_parser). It is not HA specific so it would be beneficial for others as well. Please mention me or this topic in that case.

  • Adding a hourly gas usage derivative should be trivial. I will try if I can get to it soon.

  • What is the kindergarten.nl (site doesn’t work) integration? I think intergration with dsmr-reader should not become part of this component as that would bloat the component beyond it intended purpose. But maybe I can help point you in the right direction on how to make integrations with HA an dsmr-reader.

Also kudos to https://github.com/ndokter/ who wrote dsmr_parser.

79

Can you try this: https://raw.githubusercontent.com/aequitas/home-assistant/e15e1db170bce0b603412cc838c14a6784920b97/homeassistant/components/sensor/dsmr.py

There is a new entity with m3/h. It should start showing a value after at most an hour.

80

I have a Kaifa meter, as I didn’t buy the meter myself, but was pushed by my power infrastructure company, I didn’t get a manual.

But so far the script ran fine.