Get your smart electric, water and gas meter scm readings into home assistant with a RTL-SDR

mmiller7 · May 9, 2021, 3:19am

For another RTL-based addon RTL_433 I looked at one of the sensor values and if it hadn’t changed in an interval I expected it should, I restart the addon. I used a template-sensor on the last_updated attribute and made an educated guess if its gone more than X time in my case that likely means I missed some data and it is stuck as an automation trigger.

I’m using a separate Linux system that pre-dates my home automation to run RTLAMR and push it into MQTT over my network…so I have a number of watchdogs on the services that I don’t think would be possible with HomeAssistant the way it uses containers.

rlowens · May 9, 2021, 3:42am

I’m running RTLAMR in a separate VM from HomeAssistant. I just need help creating those watchdogs, if you don’t mind.

mmiller7 · May 9, 2021, 12:50pm

Sure.

I run my watchdog with a cron-job (crontab) on the ‘root’ user

*/5 * * * * /root/rtlamr_watchdog.sh >/dev/null 2>&1

Here’s the contents of my script for the watchdog (note: requires lsof utility installed):

#!/bin/bash
#
# This file is designed to monitor for a failure in soundmodem or aprx and restart the process automatically
# Save this file to /root/aprx_watchdog.sh
#
# Crontab entry
# */5 * * * * /root/rtlamr_watchdog.sh >/dev/null 2>&1

# watchdog if rtl_tcp crashes
/bin/systemctl is-active --quiet rtl_tcp || (
        /bin/echo 'Service rtl_tcp failed, restarting rtl_tcp and rtlamr . . .' | /usr/bin/wall
        /usr/sbin/service rtl_tcp restart
        sleep 5
        /usr/sbin/service rtl_amr restart
        sleep 30
)

# watchdog if rtlamr crashes
/bin/systemctl is-active --quiet rtlamr || (
        /bin/echo 'Service rtlamr failed, restarting . . .' | /usr/bin/wall
        /usr/sbin/service rtlamr restart
        sleep 30
)

# watchdog if rtl_tcp networking fails
/usr/bin/lsof -i | grep rtl_tcp | grep LISTEN > /dev/null || (
        /bin/echo 'Network socket rtl_tcp failed, restarting rtl_tcp and rtlamr . . .' | /usr/bin/wall
        /usr/sbin/service rtl_tcp restart
        sleep 5
        /usr/sbin/service rtl_amr restart
        sleep 30
)

# watchdog if rtlamr networking fails
/usr/bin/lsof -i | grep rtlamr | grep ESTABLISHED > /dev/null || (
        /bin/echo 'Network socket rtlamr failed, restarting . . .' | /usr/bin/wall
        /usr/sbin/service rtlamr restart
        sleep 30
)

Here is what I used to build the init-startup scripts for rtl_tcp and rtlamr, note you would probably need to tweak what it runs since mine was aimed a PHP script (which writes to my sqlite app, and also throws out over MQTT)
You will want services for both rtl_tcp and rtlamr separately if you use my watchdog as-is. Basically I clone someone else’s template and change some fields…I automated it for rtl_tcp at least.

github.com

mmiller7/power_meter_graph/blob/0cd64d98be426eb9bb6e6e1491132d718203b2c8/install_rtltcp_startup_script.sh

#!/bin/bash

#Install init.d script template
echo '**** Fetching init.d script template ****'
git clone https://github.com/fhd/init-script-template.git
cd init-script-template
echo '**** Configuring aprx init.d from template ****'
cp template rtl_tcp
sed -i 's|cmd=""|cmd="/usr/local/bin/rtl_tcp"|g' rtl_tcp
sed -i 's/user=""/user="rtl_tcp"/g' rtl_tcp
sed -i 's/# Required-Start:    $remote_fs $syslog/# Required-Start:    $remote_fs $syslog rtl_tcp/g' rtl_tcp
sed -i 's/# Provides:/# Provides: rtl_tcp/g' rtl_tcp
sed -i 's/# Description:       Enable service provided by daemon./# Description:       Starts rtl_tcp daemon/g' rtl_tcp

echo '**** Installing aprx init.d ****'
cp rtl_tcp /etc/init.d/

# Give it a limited user account to run under
useradd -r -s /sbin/nologin -M rtl_tcp

This file has been truncated. show original

At this point I don’t have my repository fully updated but that should give you the bits I use.

Hopefully that helps!

DayRunner131 · June 6, 2021, 7:31pm

Help Requested.

After running this flawlessly on a raspi zero w for nine months, I shut it down to mount the raspi and reroute some wires. Now I’m not getting any updates to HA. I’m using Mosquito MQTT on the HA side.

What I’m looking for is ways to monitor each process for errors. My concern is that I may have messed up the coax wire into the SDR.

How can I monitor what rtlamr is seeing? How do I see if amrscm2mqtt is pushing a MQTT message and see if the issue is on the HA side? I do not see any MQTT logins like I used to in the HA logs.

Any troubleshooting help would be appreciated. Thanks!

biochemguy · July 12, 2021, 2:23am

My RTL-SDR has fried on me once and it could not read anything. I am guessing that it was a cooling problem. You can always see the status of the service amridm2mqtt by typing in the console using

sudo systemctl status amridm2mqtt

However, when my SDR died, I confirmed that it was not working by trying to tune it to FM radio on my windows PC using one of many free softwares

As far as looking at MQTT signals, in Home Assistant, you can always listen to all MQTT clients by going in the Configuration > Intergrations > MQTT > Configure. Then in the listen dialogue, type # and listen… It will give every message that Home Assistant can receive.

navet · August 5, 2021, 9:31pm

Just to advise this is working great with the new Energy Monitoring in 2021.8.

Had to add a few more bits to the code per this thread - Energy: Not showing the expected entities for consumption - Configuration - Home Assistant Community (home-assistant.io)

  - platform: mqtt
    state_topic: "utilityreading/100855590/reading"
    name: "House Power Used"
    unit_of_measurement: kWh
    device_class: energy
    state_class: measurement
    value_template: "{{ value |float / 1000 }}"

homeassistant:
  # Customization file
  customize: !include customize.yaml
  customize_glob:
    sensor.*_used:
      last_reset: '1970-01-01T00:00:00+00:00'
      device_class: energy
      state_class: measurement

a3a · August 27, 2021, 5:54pm

I forked the original repo and rewrote the monitor to scan for “all” ERT types, so it includes SCM, SCMplus, IDM.

In addition to that:

The “state” value is still the base consumption in kWh but I added all attribute data from the SCM and IDM responses, so you can see extra information (see below for examples)
Messages are posted with ‘retain’ flag enabled so that the meter information is immediately available to HASS on restart
Messages are published to topics in more unique namespaces for easier filtering
Messages are published to an availability topic so that HASS can update accordingly when the service is up or down
A last_reset hack value is posted to workaround the issues with MQTT sensor and last_reset.

See the fork here: GitHub - TonyApuzzo/amridm2mqtt: runs rtlamr to read IDM power meter data and send to MQTT broker<

Example reading an SCM Meter:

state_class: measurement
last_reset: '1970-01-01T00:00:00+00:00'
ChecksumVal: 55361
Consumption: 74504
ID: XXXXXXXX
TamperEnc: 0
TamperPhy: 0
Type: 4
unit_of_measurement: kWh
friendly_name: Electric Meter
device_class: energy

Example reading an IDM Meter:

state_class: measurement
last_reset: '1970-01-01T00:00:00+00:00'
ApplicationVersion: 2
AsynchronousCounters: 22389
ConsumptionIntervalCount: 0
DifferentialConsumptionIntervals:  [ 1, 2, 1, 0, 2, 1, 2, 1, 0, 6, 10, 10, 10, 10, 10, 12, 11, 11, 10, 10, 10, 10, 11, 10, 11, 12, 11, 12, 12, 11, 12, 10, 13, 15, 11, 10, 11, 10, 11, 10, 7, 2, 1, 2, 1, 1, 0 ]
ERTSerialNumber: YYYYYYYY
ERTType: 7
HammingCode: 198
LastConsumptionCount: 2673708
ModuleProgrammingState: 204
PacketCRC: 60717
PacketLength: 92
PacketTypeID: 28
PowerOutageFlags: opAwAKEg
Preamble: 1431639715
SerialNumberCRC: 52453
TamperCounters: FoIhbUgS
TransmitTimeOffset: 346
unit_of_measurement: kWh
friendly_name: Electric Meter IDM
device_class: energy

Definitions for the above:

sensor:
  - platform: mqtt
    state_topic: "amr/reading/SCM/4/XXXXXXXX/message"
    name: "Electric Meter"
    unique_id: electric_meter_01
    unit_of_measurement: kWh
    device_class: energy
    state_class: measurement
    availability_topic: amr/status/availability
    last_reset_topic: amr/status/last_reset
    value_template: "{{ value_json.Message.Consumption }}"
    json_attributes_template: "{{ value_json.Message | tojson }}"
    json_attributes_topic: "amr/reading/XXXXXXXX/message"
    
  - platform: mqtt
    state_topic: "amr/reading/IDM/7/YYYYYYYY/message"
    name: "Electric Meter IDM"
    unique_id: electric_meter_02
    unit_of_measurement: kWh
    device_class: energy
    state_class: measurement
    availability_topic: amr/status/availability
    last_reset_topic: amr/status/last_reset
    value_template: "{{ value_json.Message.Consumption }}"
    json_attributes_template: "{{ value_json.Message | tojson }}"
    json_attributes_topic: "amr/reading/IDM/7/YYYYYYYY/message"

noebl1 · September 4, 2021, 2:07pm

I put in a feature request for the Energy dashboard for those using rtlamr to be able to use the ccf reading from this as part of the gas consumption usage for those interested:

I could not find a way currently to get it to show up as an option for consumption.

a3a · September 7, 2021, 5:16pm

Note that with the 2021.9 release, using last_reset_topic is deprecated as state_class: total_increasing is available to handle metering logic automatically.

Using my fork, you should be able to use the following definition of an MQTT sensor for gas meters:

packages/energy.yaml:

sensor:
  - platform: mqtt
    state_topic: !secret GAS_METER_01_TOPIC
    name: "Gas Meter"
    unique_id: gas_meter_01
    unit_of_measurement: m³
    device_class: gas
    state_class: total_increasing
    availability_topic: amr/status/availability
    # My meter reads in cubic feet, convert to cubic meters
    value_template: "{{ value_json.Message.Consumption | float * 0.02832 }}"
    json_attributes_template: "{{ value_json.Message | tojson }}"
    json_attributes_topic: !secret GAS_METER_01_TOPIC

secrets.yaml:

GAS_METER_01_TOPIC: 'amr/reading/SCM/2/32109876/message'

It turns out that my home gas meter only broadcasts values once per month when the utility company pings it, so it is hard to figure out if it is fully working or not. In my installation, the above sensor doesn’t show up in the ᴀᴅᴅ ɢᴀs sᴏᴜʀᴄᴇ dialog, so I am not sure if I have to wait until the next reading goes out in a month or if there is something else wrong.

EDIT: corrected state_class value in example.
EDIT2: added conversion from cubic feet to cubic meters

noebl1 · September 7, 2021, 7:59pm

I think part of it too is m³ is supported, but ccf is not for gas from what I recall from the release stream they did, but I’m not 100% on this.

EDIT: Supported for the Energy Dashboard that is

a3a · September 7, 2021, 11:04pm

yeah, my mqtt sensor said m³ but the consumption value provided by my meter is in cubic feet, I’ve edited my post above to include conversion between cf and m³.

My paper bill shows both therms and ccf, with the ccf number being 1/100 of what my meter reports. Annoyingly, the meter reading is only shown on my paper bill. The online statement that Xcel keeps pushing me to use only shows therms. It would be much harder to figure out what the actual values were if I was using online only billing.

mdleal · September 17, 2021, 2:46pm

I have it setup like your edited post but still can’t add it under the energy dashboard.

a3a · September 18, 2021, 12:32am

Sadly I am in the same boat. I have been trying to get it to show up without success.

Apologies for excessive confidence above, I was ASSuMEing that mine would show up once I got some MQTT messages from the gas meter, but that isn’t the case.

UPDATE: One month later and now my gas meter defined exactly as before is now showing up in the Energy Configuration page. Not sure what triggered it to start working, but I notice that the selection is now available after installing 2010.10.6

makeitworktech · September 22, 2021, 11:33am

Hi @a3a this is super cool. This question is coming from someone with very little Raspberry Pi or Linux experience. Am I right to assume that all the installing steps (rtl-sdr package, gi, pip, paho-mqtt package, etc.) are done on a RaspBerry Pi running Linux? I see the repo says “Tested under Raspbian GNU/Linux 9.3 (stretch)”. Forgive me if the question is simple. I’d like to set this up myself, and maybe make a little video on how to do it for people who have only used a Pi for Home Assistant.

jeffeb3 · October 6, 2021, 8:06pm

Is there a list of devices that have been confirmed to not work?

I just got a new meter (Colorado, USA) and it is an itron “Gen 5 Riva” (model: G5R1 and “TYPE R2SID”). The brocure is here and the fcc spec is here. It says things like 2.4GHz and the OS is Linux (I like Linux, but that tells me it may be getting too smart). It also has the AMI 900MHz band, so maybe it is doing the mesh networking and the AMI broadcast. My guess is that it could be a config option for the installer.

It would be nice if someone had already done all the legwork to be able to report that, no, it doesn’t work. Instead, I am hoping it is a new kind of meter that no one has tried. I ordered and SDR anyway. Maybe I’ll find a use for it if this doesn’t work.

SpicyArtichoke · October 7, 2021, 12:42am

This resolved the velow errors for me!
amridm2mqtt.service: Failed to execute command: Permission denied
amridm2mqtt.service: Failed at step EXEC spawning /opt/amridm2mqtt/amrscm2mqtt: Permission denied

DiamondCocktail · October 7, 2021, 3:04am

I saw some people noting about needing multiple frequencies (433/900). I don’t have it in front of me, but rtl_433 has parameter to hop between different frequencies on a time interval. Should be in the docs.

a3a · October 18, 2021, 8:32pm

Yes, all the AMR RTL stuff is done on a Raspberry Pi (separately from my HA install) it communicates with HA via MQTT. I used a Raspberry Pi 3b+. The 4 would be a better choice if you’re purchasing a new device as the rtlamr process is fairly heavy, consuming 25%-50% of the CPU on the 3b+.

Also note that something finally freed up and with the very latest HA 2021.10.6, my gas meter defined exactly as above is now showing up in HA energy configuration page.

a3a · October 18, 2021, 8:44pm

The original author of the RTL-SDR AMR project has a form to submit working/non-working meters.

So if your meter isn’t already in the list, and you can later confirm that it does or doesn’t work, you can update the form he linked.

See: https://github.com/bemasher/rtlamr/blob/3965b5c5239c76bbc813b82d27fcb2c6b187a657/meters.md

jeffeb3 · October 20, 2021, 5:47pm

I only see the compatible meters, not any incompatible ones. So with a meter that seems to be a new model, how do you know if it is confirmed to not work, or a meter that no one has tried?