Get your smart electric, water and gas meter scm readings into home assistant with a RTL-SDR

Tags: #<Tag:0x00007f3f1ac7f438>

Hi,

Just wanted to share this project, in case someone wanted to do the same. I added my utilities smart meters into home assistant, so I can monitor my current usage, weekly usage and more.

I bought a NooElec NESDR Nano 2+, plugged it in my raspberry pi 3B+ and proceeded with programming it to read the meters and send the data through MQTT to home assistant.

Before buying a dongle,If you want to know if your meter can be read this way, here is the list of meters tested with rtlamr. Other types of smart meter might be encrypted and will not work.

I tried a few ways to send the info through MQTT and found that this way from ragingcomputer was the best for me to get it working. I had to modify the python script because this one was setup to read idm data from smart meters and while my power meter was sending data with this format, my gas meter was sending scm data. I decided to change it, so it would send the scm data from both to mqtt. I also added a delay to start rtl_tcp as I had a problem with it starting it during boot.

Here is my process:

rtl-sdr package

Install RTL-SDR package

sudo apt-get install rtl-sdr

Set permissions on rtl-sdr device

sudo nano /etc/udev/rules.d/rtl-sdr.rules

SUBSYSTEMS=="usb", ATTRS{idVendor}=="0bda", ATTRS{idProduct}=="2838", MODE:="0666"

Prevent tv tuner drivers from using rtl-sdr device

sudo nano /etc/modprobe.d/rtl-sdr.conf

blacklist dvb_usb_rtl28xxu

git

sudo apt-get install git

pip3 and paho-mqtt

Install pip for python 3

sudo apt-get install python3-pip

Install paho-mqtt package for python3

sudo pip3 install paho-mqtt

golang & rtlamr

Install Go programming language & set gopath

sudo apt-get install golang

If only running go to get rtlamr, just set environment temporarily with the following command

export GOPATH=$HOME/go

Install rtlamr https://github.com/bemasher/rtlamr

go get github.com/bemasher/rtlamr

To make things convenient, copy rtlamr to /usr/local/bin

sudo cp ~/go/bin/rtlamr /usr/local/bin/rtlamr

Install

Clone Repo

Clone repo into opt

cd /opt

sudo git clone https://github.com/ragingcomputer/amridm2mqtt.git

Configure

Copy template to settings.py

cd /opt/amridm2mqtt

sudo cp settings_template.py settings.py

Edit file and replace with appropriate values for your configuration

sudo nano /opt/amridm2mqtt/settings.py

Install Service and Start

Create a new file amrscm2mqtt:

sudo nano amrscm2mqtt

Paste this code and save:

'#!/usr/bin/env python3
'''
Runs rtlamr to watch for IDM broadcasts from power meter. If meter id
is in the list, usage is sent to 'readings/{meter id}/meter_reading'
topic on the MQTT broker specified in settings.

WATCHED_METERS = A Python list indicating those meter IDs to record and post.
MQTT_HOST = String containing the MQTT server address.
MQTT_PORT = An int containing the port the MQTT server is active on.

'''
import subprocess
import signal
import sys
import time
import paho.mqtt.publish as publish
import settings

# uses signal to shutdown and hard kill opened processes and self
def shutdown(signum, frame):
    subprocess.call('/usr/bin/pkill -9 rtlamr', shell=True)
    subprocess.call('/usr/bin/pkill -9 rtl_tcp', shell=True)
    subprocess.call('/usr/bin/pkill -9 amridm2mqtt', shell=True)
    sys.exit(0)

signal.signal(signal.SIGTERM, shutdown)
signal.signal(signal.SIGINT, shutdown)

auth = None

if len(settings.MQTT_USER) and len(settings.MQTT_PASSWORD):
        auth = {'username':settings.MQTT_USER, 'password':settings.MQTT_PASSWORD}

# send data to MQTT broker defined in settings
def send_mqtt(topic, payload,):
    try:
'#!/usr/bin/env python3
'''
Runs rtlamr to watch for IDM broadcasts from power meter. If meter id
is in the list, usage is sent to 'readings/{meter id}/meter_reading'
topic on the MQTT broker specified in settings.

WATCHED_METERS = A Python list indicating those meter IDs to record and post.
MQTT_HOST = String containing the MQTT server address.
MQTT_PORT = An int containing the port the MQTT server is active on.

'''
import subprocess
import signal
import sys
import time
import paho.mqtt.publish as publish
import settings

# uses signal to shutdown and hard kill opened processes and self
def shutdown(signum, frame):
    subprocess.call('/usr/bin/pkill -9 rtlamr', shell=True)
    subprocess.call('/usr/bin/pkill -9 rtl_tcp', shell=True)
    subprocess.call('/usr/bin/pkill -9 amridm2mqtt', shell=True)
    sys.exit(0)

signal.signal(signal.SIGTERM, shutdown)
signal.signal(signal.SIGINT, shutdown)

auth = None

if len(settings.MQTT_USER) and len(settings.MQTT_PASSWORD):
        auth = {'username':settings.MQTT_USER, 'password':settings.MQTT_PASSWORD}


def send_mqtt(topic, payload,):
    try:
        publish.single(topic, payload=payload, qos=1, hostname=settings.MQTT_HOST, port=settings.MQ$
    except Exception as ex:
        print("MQTT Publish Failed: " + str(ex))

time.sleep(30)

# start the rtl_tcp program
rtltcp = subprocess.Popen([settings.RTL_TCP + " > /dev/null 2>&1 &"], shell=True,
    stdin=None, stdout=None, stderr=None, close_fds=True)

time.sleep(5)

# start the rtlamr program.
rtlamr = subprocess.Popen([settings.RTLAMR,
    '-msgtype=scm',
    '-format=csv'], stdout=subprocess.PIPE)

while True:

    try:
        # rtlamr's readline returns byte list, remove whitespace and convert to string
        amrline = rtlamr.stdout.readline().strip().decode()
        # split string on commas
        flds = amrline.split(',')

        if len(flds) != 9:

            # proper SCM results have 9 fields
            continue

        # make sure the meter id is one we want
        meter_id = int(flds[3])
        if len(settings.WATCHED_METERS) and meter_id not in settings.WATCHED_METERS:
            continue

        # get some required info: current meter reading, current interval id, most recent interval $
        read_cur = int(flds[7])

        current_reading_in_kwh = (read_cur * settings.WH_MULTIPLIER) / 1000

        send_mqtt(
            'readings/' + str(meter_id) + '/meter_reading',
            '%s' % (current_reading_in_kwh)
        )

    except:
        time.sleep(2)

Then edit the service configuration into to point to this new file

ssudo nano /opt/amridm2mqtt/amridm2mqtt.systemd.service

Modify this line ExecStart=/opt/amridm2mqtt/amridm2mqtt

so it points to the new file:

ExecStart=/opt/amridm2mqtt/amrscm2mqtt

then copy service configuration:

sudo cp /opt/amridm2mqtt/amridm2mqtt.systemd.service /etc/systemd/system/amridm2mqtt.service

Refresh systemd configuration

sudo systemctl daemon-reload

Start amridm2mqtt service

sudo service amridm2mqtt start

Set amridm2mqtt to run on startup

sudo systemctl enable amridm2mqtt.service

Configure Home Assistant

To use these values in Home Assistant,

sensor:
  - platform: mqtt
    state_topic: "readings/12345678/meter_reading"
    name: "Power Meter"
    unit_of_measurement: kWh

  - platform: mqtt
    state_topic: "readings/12345678/meter_reading"
    name: Gas Meter"
    unit_of_measurement: M3

Here is the settings for current value. Since my meters send data approx. every 5 minutes, I did an average based on a 10 minute average. I used these settings because my current reading would do a false spike when home assistant was restarted, as the statistics minimum value would reset to zero:

sensors:
  - platform: template
    sensors:
      current_power_consumption:
        value_template: >
          {% if states.sensor.power_stats_mean.attributes.min_value | float == 0 %}0
          {% elif states.sensor.power_meter.state | float <= states.sensor.power_stats_mean.attributes.min_value | float %}0
          {% else %}
          {{((states.sensor.power_meter.state | float) - (states.sensor.power_stats_mean.attributes.min_value | float)) * 1000 / 10 * 60 | max (0) | round(2) }}
          {% endif %}

        unit_of_measurement: 'W'
        friendly_name: "Average Current Power Consumption (10 min)"
        
  - platform: template
    sensors:
      current_gas_consumption:
        value_template: >
          {% if states.sensor.gas_stats_mean.attributes.min_value | float == 0 %}0
          {% elif states.sensor.gas_meter.state | float <= states.sensor.gas_stats_mean.attributes.min_value | float %}0
          {% else %}
          {{((states.sensor.gas_meter.state | float) - (states.sensor.gas_stats_mean.attributes.min_value | float)) * 1000 / 10 | max (0) | round(2) }}
          {% endif %}
        unit_of_measurement: 'cdm/min'
        friendly_name: "Average Current Gas Consumption (10 min)"

 
 
  - platform: statistics
    name: gas_stats
    entity_id: sensor.gas_meter
    max_age:
      minutes: 10
    friendly_name: "Gas Stats"
    
  - platform: statistics
    name: power_stats
    entity_id: sensor.power_meter
    max_age:
      minutes: 10
    friendly_name: "Power Stats"

For the current daily usage:

sensors:
  - platform: template
    sensors:
      todays_power_consumption:
        value_template: >
          {{ ((states.sensor.power_meter.state | float) - (states.input_number.midnight_power_consumption.state | float)) | max (0) | round(2) }}
        unit_of_measurement: 'kW'
        friendly_name: "Today's Power Consumption"
        
  - platform: template
    sensors:
      todays_gas_consumption:
        value_template: >
          {{ ((states.sensor.gas_meter.state | float) - (states.input_number.midnight_gas_consumption.state | float)) | max (0) | round(2) }}
        unit_of_measurement: 'M3'
        friendly_name: "Today's Gas Consumption"

input_number:
  midnight_power_consumption:
    name: Midnight Power Level
    min: 0
    max: 999999999
 
  midnight_gas_consumption:
    name: Midnight Gas Level
    min: 0
    max: 999999999

automation:
- id: '1527474342231'
  alias: Power Consumption start of the day
  trigger:
    platform: time
    at: '00:00:00'
  action:
    - service: input_number.set_value
      data_template:
        entity_id: input_number.midnight_power_consumption
        value: "{{ states.sensor.power_meter.state }}"
    - service: input_number.set_value
      data_template:
        entity_id: input_number.midnight_gas_consumption
        value: "{{ states.sensor.gas_meter.state }}"

Testing

Assuming you’re using mosquitto as the server, and your meter’s id is 12345678, you can watch for events using the command:

mosquitto_sub -t "readings/12345678/meter_reading"

Or if you’ve password protected mosquitto

mosquitto_sub -t "readings/12345678/meter_reading" -u <user_name> -P <password>

Hope this helps someone else with their setup. It took me a while to get it working.

9 Likes

Interesting, how do I know if my meter is sending data. I already have a SDR, I use rtl 433 to read various accurite sensors and diy temp device I made.

Depending on your setup, you might already have some packages installed. Basically you would need to follow this setup (apart from paho-mqtt) until rtlamr is installed. Once that is done, open two terminals and in the first one run rtl_tcp and in the other run rtlamr. You should see a stream of data from the meters around you. Find the ones that match your meter number and you are set. More details can be found here https://github.com/bemasher/rtlamr.

I was actually wondering if it was possible to read the meters while reading weather data or other things like FM radio without overloading the system?

it looks like AMR data is coming over 900mhz range while my weather stuff is 433mhz, only solution I can think of is another SDR . My weather stuff is running in docker and i notice when I run rtl_tcp and rtlamr in two different terminals I get out put that look like this

{Time:2019-01-24T09:27:27.759 SCM:{ID:34528191 Type:12 Tamper:{Phy:00 Enc:00} Consumption: 417372 CRC:0x34C0}}
{Time:2019-01-24T09:27:30.535 SCM:{ID:38392240 Type:12 Tamper:{Phy:02 Enc:00} Consumption: 406242 CRC:0x576E}}

with a lot more ID , so I am guessing I am picking up the neighbors as well and have to figure out my id. From the type it looks to be gas, but I have not looked at the meters to figure it out.

I think the best solution for both 433mh weather and 900mhz amr is two SDR antena , if I can dockerize this it will work great. I hope to find time to play with it over the weekend.

BTW this is great find, new ideas to tinker with.

1 Like

I was not sure if two programs could control the SDR at the same time for different frequencies. I did not try it, but I was debating getting a weather station if it can read both frequencies at once. Logically, doesn’t the SDR receive a big range of frequency and the software decides what to do with it? If the dongle can communicate to both programs at the same time, it could work.

You did find gas meter data (Type:12). Your meter ID should be written on your meter or maybe on your bill (Account number?).

I have not been out to look at the meter. I actually tried the arm software from remotely and it works but I had to shut down the rtl 433 docker container that I built first,

I can look at the amr 2 mqtt software ymaybe incorporate it with my rtl433 to mqtt code that is in python and see if I can make them alternate polling.

It might be easier just to get another sdr

I tried this awhile ago and at least in my area (California), the meters are encrypted so the data can’t be read. Be sure to check the rtlamr site to see if the meter is supported before you buy a radio.

That’s too bad that you could not read it…

I placed a link to show the supported meters in the start of the instructions. I now updated it so that it’s in bold.

Great tutorial. I am working though it. I am able to read my water and gas meter wby using rtlamr, but the problem I am having is registering the service on my raspberry pi. Every time I try to start the service, I get the error:

amridm2mqtt.service: Failed at step EXEC spawning /opt/amridm2mqtt/amrscm2mqtt: Permission denied

Any idea what this would be caused by?

Thanks

I am guessing that the readings you are getting are from rtlamr and not amrscm2mqtt script. You can make sure that the amrscm2mqtt script is running fine by running it in one terminal with the rtl-tcp running in another terminal (started first) and a third terminal with mosquitto (or other broker) where you would receive the mqtt data.

When you modified the service file,it should not be blank. I just noticed that in my tutorial, I had not put the correct link there (I will fix it). It should be:

sudo nano /opt/amridm2mqtt/amridm2mqtt.systemd.service instead of sudo nano systemd.service

There you should see the line ExecStart=/opt/amridm2mqtt/amridm2mqtt and modify it to :

ExecStart=/opt/amridm2mqtt/amrscm2mqtt

If it still doesn’t work and the original service file works (the one that points to the amridm2mqtt), you could always replace the contents of this file. with the contents of the amrscm2mqtt file

Finally, after doing a quick search for Failed at step EXEC spawning - Permission denied, I am not sure if this could help, but you might simply need to change the permissions of the file by doing sudo chmod +x /opt/amridm2mqtt/amrscm2mqtt

Hope this helps!

just curious are you in the states? I am very interested in doing this with my home-assistant, are the readings identical? does this work with therms?

damn, this sucks so I assume you weren’t able to get it to work, do you have pge?

I am using lovelace UI. The readings are good and match my utilities. I can see in my power readings when the dryer is on or when a baseboard heater has turned on with a graph that shows a few days using the current average power. With a recent update of Home Assistant, they created a new utility meter component that might help someone with calculating daily power, monthly power and costs instead of using my configurations.

If you are interested in doing this project, just make sure that your meters are compatible…

I had SCE and then Glendale W&P- both were encrypted. Take a pic of your meter and compare it withe the rtamr repo page - there is a list of supported meters there.

Can confirm here in SoCal for electrical I’m also on SCE with a Zigbee-based smart meter.

Thanks @biochemguy, I will try this out in the next day when I have some time. I did some more research tonight as well and I think I might have the solution.

My water and gas meter is compatible with the rtlamr library, its amazing how many neighbours I pick up doing a 10 minute monitor. Unfortunately my power meter is also zigbee encrypted, however there is a device made by Rainforest automation that is compatible with my meter. There are some other posts about integrating it.

1 Like

if someone is interested, here is my dashboard using those components. I also added calculations that use the cost for the last 30 days of utility use. This uses the scrape sensor from home assistant to get the current rates for my utilities, as the gas will fluctuate. It’s not perfect, but it’s good enough for me…

Oh and yes, it was a cold month, lol!

2 Likes

can you measure in therms?

according to PGE’s website, the data isn’t encrypted until it his the network nodes (their wireless access points), going to try it out anyways, it’ll only be a loss of $35 if I can’t get it to work

Does it measure gas or just electric?