Power monitoring with an XTM18S and MQTT


#1

Power monitoring with an XTM18S and MQTT

Objective

The main aim of this project is to acquire the electric power consumption of a single phase two wire active line and send the results to HASS via MQTT with the aid of an ESP8266 device.

(My usual) Disclaimer: I’m a computer hobbyist, self-learner, and with proficiency in copy-paste tool. What I describe here has worked for me but YMMV. As you may have guessed, my mother language is not English so sorry for any vocabulary inaccuracy, and so on.

(A very important) Disclaimer (in this project): This project deals with high voltages. It is dangerous. Use safely and responsibly. I am not responsible for any damages caused by the building of this device.

Introduction

In my opinion, electricity power monitoring is one of the most important things to do in a domotic system. I say this because I conceive a domotic system as a solution not only to perform automations (and make our lives more comfortable) but to save costs (and reduce our CO2 fingerprint).

There are some projects related to power monitoring. I tend to classify them in two categories:

  1. Cheap DIY solutions, such as this one.
  2. Finished product solutions: I’m not going to link any commercial alternatives but googling items such as “energy meter” will give you a lot of options.

Not being a guy with a lot of spare time or the needed knowledge, the DIY approach is sometimes difficult for me. On the other side, commercial solutions can be expensive or just don’t let us improve our creative thinking. This project falls in the middle of the two categories mentioned before.

Finally, I would like to mention this project. They have a lot of resources and I’m considering to support their project by buying one of the solutions they shell. I just need some time to study the solution that better suits to me.

The hardware

I have just coupled two cheap devices and this operation does not involve any soldering tasks. These two devices are:

  1. An XTM18S single phase electronic DIN rail active energy meter. You can find the product for about 12 € in ebay here. It has 6 connections: 2 line IN (which you connect to your mains), 2 line OUT (which you connect to your electric load) and 2 additional outputs (where you get a pulsed signal that we will use to get the electric consumption). The device also includes a display that shows the accumulated energy consumption and an LED with three states: no color (when the IN line is disconnected), green color (when the IN line is connected) and a red color blink (every time the electric consumption achieves a certain value).
  2. A Wemos D1 mini, which is a very well known cheap ESP8266 device with Wifi capabilities. This is the device that we will use to connect to the XTM18S in order to count the pulses that will be later sent to our MQTT broker via Wifi.

The electric connections are fairly easy to setup. On one extreme of the XTM18S you can find the numbers 1 and 4, which correspond to the IN line conections. In the attached picture it is the cable that is connected to the wall mains. On the opposite side you’ll find 4 connections. The bottom ones, marked as 3 and 6, are the OUT lines and must be connected to the electric appliance (a hair-dryer in the attached picture).

In the upper part of this side you’ll see the connections 20 and 21. I’ve connected with a standard jumper wire the 20 port to the D6 pin in the Wemos D1 mini while the 21 port to the ground pin. Finally, you have to power the Wemos D1 mini. In these first tests I’ve used a power-bank but in its final placement I’ll have to switch it on to mains since ESP8266 devices are quite power hungry.

However, before setting up the hardware, you have to upload the software to the ESP8266 so, please, check the next section first.

The software

Three pieces of software are described in this section:

Step 1. Below you’ll find the code of the INO file that I uploaded to the Wemos D1 mini using the Arduino IDE. There are wonderful guides in the Internet explaining how to do this. I used this one.
You have to update the information in the code regarding your SSID and MQTT server details. You can also change the PIN that is reading the pulses.
If I’m not wrong all the pins of the Wemos D1 mini (except D0) allow interruptions but you’ll have to check this if you are using other ESP8266 device. I think the code is self-explanatory but, please, ask if you have questions.

/**
 * This script is based on these sources:
 *
 * To count pulses using interruptions:  https://github.com/mysensors/MySensors/blob/master/examples/EnergyMeterPulseSensor/EnergyMeterPulseSensor.ino
 * To connect to Wifi and publish MQTT messages:  https://github.com/knolleary/pubsubclient/blob/master/examples/mqtt_esp8266/mqtt_esp8266.ino
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * version 2 as published by the Free Software Foundation.
 *
 *******************************
 *
 * DESCRIPTION
 * Use this sensor to measure kWh and Watt of your house meter.
 * You need to set the correct pulsefactor of your meter (pulses per kWh).
 * Reports every SEND_FREQUENCY miliseconds: pulses counting, kWh and Watt to different MQTT topics.
 *
 */

#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.

/************************* WiFi Access Point *********************************/

const char* ssid = "yourssid";
const char* password = "yourpassword";

/**************************** MQTT Broker ************************************/

const char* mqtt_server = "mqttserverip"; // example: "192.168.0.8"
const char* mqtt_username = "yourmqttuser";
const char* mqtt_password = "yourmqttpass";
const char* mqtt_topic_watt = "ESP-energy-01/watt";
const char* mqtt_topic_kwh = "ESP-energy-01/kwh";
const char* mqtt_topic_pulse = "ESP-energy-01/pulse";

#define DIGITAL_INPUT_SENSOR 12 // The digital input you attached S0+ D6 in Wemos D1 mini
#define PULSE_FACTOR 1000       // Nummber of pulses per kWh of your meeter
#define MAX_WATT 10000          // Max watt value to report. This filters outliers.

unsigned long SEND_FREQUENCY = 20000; // Minimum time between send (in milliseconds)
double ppwh = ((double)PULSE_FACTOR)/1000; // Pulses per watt hour
volatile unsigned long pulseCount = 0;
volatile unsigned long lastBlink = 0;
volatile unsigned long watt = 0;
unsigned long oldWatt = 0;
double oldKwh;
unsigned long lastSend;

WiFiClient espClient;
PubSubClient client(espClient);

long lastMsg = 0;
char msg[50];
char wattString[6];
char kwhString[6];
char pulseCountString[6];

void setup_wifi() {
  delay(10);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

// Setup a MQTT subscription
void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
}

void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("ESP8266Client", mqtt_username, mqtt_password)) {
      Serial.println("connected");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}

void setup()
{
  Serial.begin(115200);
  setup_wifi();
  client.setServer(mqtt_server, 1883);
  client.setCallback(callback);
  // Use the internal pullup to be able to hook up this sketch directly to an energy meter with S0 output
  // If no pullup is used, the reported usage will be too high because of the floating pin
  pinMode(DIGITAL_INPUT_SENSOR,INPUT_PULLUP);
  attachInterrupt(digitalPinToInterrupt(DIGITAL_INPUT_SENSOR), onPulse, RISING);
  lastSend=millis();
}

void loop()
{
    if (!client.connected()) {
        reconnect();
    }
    client.loop();

    unsigned long now = millis();
    // Only send values at a maximum frequency
    bool sendTime = now - lastSend > SEND_FREQUENCY;
    if (sendTime) {
        // New watt value has been calculated
        if (watt != oldWatt) {
            // Check that we dont get unresonable large watt value.
            // could hapen when long wraps or false interrupt triggered
            if (watt<((unsigned long)MAX_WATT)) {
                // convert to a string with 2 digits before the comma and 2 digits for precision
                dtostrf(watt, 4, 1, wattString);
                client.publish(mqtt_topic_watt,wattString);  // Publish watt to MQTT topic
            }
            Serial.print("Watt:");
            Serial.println(wattString);
            oldWatt = watt;
            dtostrf(pulseCount, 4, 1, pulseCountString); // To Do: convert int to str, but not like this
            client.publish(mqtt_topic_pulse,pulseCountString);  // Publish pulses to MQTT topic
            double kwh = ((double)pulseCount/((double)PULSE_FACTOR));
            // convert to a string with 2 digits before the comma and 2 digits for precision
            dtostrf(kwh, 2, 2, kwhString);
            client.publish(mqtt_topic_kwh,kwhString);  // Publish kwh to MQTT topic
            oldKwh = kwh;
            lastSend = now;  // once every thing is published we update the send time
        }
    }

}

void onPulse()
{
    unsigned long newBlink = micros();
    unsigned long interval = newBlink-lastBlink;
    if (interval<10000L) { // Sometimes we get interrupt on RISING
            return;
    }
    watt = (3600000000.0 /interval) / ppwh;
    lastBlink = newBlink;
    pulseCount++;
}

Step 2. Create the MQTT sensors (in my splitted configuration setup I created this file: /home/homeassistant/.homeassistant/includes/sensors/electricity.yaml) containing:

- platform: mqtt
  name: "Electric power 01"
  state_topic: "ESP-energy-01/watt" 
  unit_of_measurement: "W"
- platform: mqtt
  name: "Electric energy 01"
  state_topic: "ESP-energy-01/kwh" 
  unit_of_measurement: "kWh"
- platform: mqtt
  name: "Electric pulses 01"
  state_topic: "ESP-energy-01/pulse" 

Step 3. Create the group to visualize in the dashboard (in my splitted configuration setup I created this file: /home/homeassistant/.homeassistant/includes/groups/electricity.yaml) containing:

electricity:
  name: Electricity
  view: yes
  entities:
    - group.xtm18s_01

xtm18s_01:
  name: 'TV consumption'
  entities:
    - sensor.electric_power_01
    - sensor.electric_energy_01
    - sensor.electric_pulses_01

Results

Main result of the described process is having a display in the HA dashboard showing the electric consumption of a given appliance. An example of this is shown is the next screenshot:

Now you can use this information to perform smart automations and notifications to try to reduce your electricity bill.

Conclusions

  • Using an XTM18S device and a Wemos D1 mini it is possible to register the electric consumption of a certain line for less than 15 €.
  • What I described here works for me up to the stage where my MQTT broker receives messages from the ESP8266 device. I have checked that pulses sent correspond to red light blinks in the XTM18S device. However, I haven’t checked if the numbers in those messages (Watts and kWh) correlate well with reality.

Final words

If I haven’t been clear enough, please ask. I’ll try to do my best to help.
It is very likely that this can be accomplished in a simpler/cheaper/safer way (remember the disclaimers). I’m ready to follow your hints (if any) and keep on learning. Thanks for reading.


#2

A nice looking project, and very well described, but please let us know how you eventually power the Wemos. The current set up would not do in my household.


#3

Thanks @gpbenton
Although I’m still thinking what will be the definitive use of this device, I have placed it inside a cupboard to measure the electric consumption of the TV. On one side I will satisfy my curiosity of how much electricity is consumed by my TV and, on the other side, I could check how much time and when it is switched on. So my current setup is as follows:

  • There is an extension cord going from the wall mains to a cupboard.
  • Inside the cupboard I have plenty of room to plug a phone charger to the extension cord powering the Wemos.
  • I have attached in and out cables to the XTM18S as shown in my first post. However, instead of having a hair-dryer, it is the TV that is plugged into the outlet of the XTM18S.

A picture of this setup is shown below.

I hope now it is more clear how I power the Wemos.
Anyway, there must be something wrong in my code since “Watts” do not go to zero when pulses do not increase, as you can detect in the figure below (TV was switched on only 5 minutes). So I have to double-check my code. Fun has just began!


#4

From what I can see, watt doesn’t change unless you get a pulse, so if no more pulses, it stays at the previous level.


#5

Yes, you are right. Thank you!
So I decided to change the philosophy in the pulses counting code.
As previously, the Wemos reports every SEND_FREQUENCY miliseconds (2 minutes in my case) messages to 3 different MQTT topics:

  • (1)- Energy (in kWh) consumed in the last SEND_FREQUENCY/1000 seconds
  • (2)- Power (in W) consumed in the last SEND_FREQUENCY/1000 seconds
  • (3)- Energy (in kWh) consumed since the device was switched on

There could be errors in (1) and (2) since pulse counting and information sending events do not take place at the same time. (3), if enough time has passed by, is as accurate as the one from the measuring device

UPDATED INO FILE

/**
 * This script is based on these sources:
 *
 * To count pulses using interruptions:  https://github.com/mysensors/MySensors/blob/master/examples/EnergyMeterPulseSensor/EnergyMeterPulseSensor.ino
 * To connect to Wifi and publish MQTT messages:  https://github.com/knolleary/pubsubclient/blob/master/examples/mqtt_esp8266/mqtt_esp8266.ino
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * version 2 as published by the Free Software Foundation.
 *
 *******************************
 *
 * DESCRIPTION
 * Use this sensor to measure kWh and Watt of your house meter.
 * You need to set the correct pulsefactor of your meter (pulses per kWh).
 * You have to update the information related to the MQTT server
 * Reports to different MQTT topics every SEND_FREQUENCY miliseconds:
 * (1)- Energy (in kWh) consumed in the last SEND_FREQUENCY/1000 seconds
 * (2)- Power (in W) consumed in the last SEND_FREQUENCY/1000 seconds
 * (3)- Energy (in kWh) consumed since the device was switched on
 * There could be errors in (1) and (2) since pulse counting and information sending events
 * do not take place at the same time. (3), if enough time has passed by, is as accurate as
 * the one from the measuring device
 */

#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.

/************************* WiFi Access Point *********************************/

const char* ssid = "your-essid";
const char* password = "your-essid-password";

/**************************** MQTT Broker ************************************/

const char* mqtt_server = "your-mqtt-server-ip";
const char* mqtt_username = "your-mqtt-user";
const char* mqtt_password = "your-mqtt-password";
const char* mqtt_topic_watt = "ESP-energy-01/watt";
const char* mqtt_topic_kwh = "ESP-energy-01/kwh";
const char* mqtt_topic_pulse = "ESP-energy-01/pulse";

#define DIGITAL_INPUT_SENSOR 12 // The digital input you attached S0+ D6 in Wemos D1 mini
#define PULSE_FACTOR 1000       // Nummber of pulses per kWh of your meeter

unsigned long SEND_FREQUENCY = 120000; // Minimum time between send (in milliseconds)
volatile unsigned long pulseCount = 0;
volatile unsigned long lastBlink = 0;
double kwh;
unsigned long lastSend;

WiFiClient espClient;
PubSubClient client(espClient);

long lastMsg = 0;
char msg[50];
char wattString[7];
char kwhString[7];
char kwhaccumString[7];

void setup_wifi() {
  delay(10);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

// Setup a MQTT subscription
void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
}

void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("ESP8266Client", mqtt_username, mqtt_password)) {
      Serial.println("connected");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}

void setup()
{
  Serial.begin(115200);
  setup_wifi();
  client.setServer(mqtt_server, 1883);
  client.setCallback(callback);
  // Use the internal pullup to be able to hook up this sketch directly to an energy meter with S0 output
  // If no pullup is used, the reported usage will be too high because of the floating pin
  pinMode(DIGITAL_INPUT_SENSOR,INPUT_PULLUP);
  attachInterrupt(digitalPinToInterrupt(DIGITAL_INPUT_SENSOR), onPulse, RISING);
  // Initialization of variables
  kwh = 0;
  lastSend=millis();
}

void loop()
{
    if (!client.connected()) {
        reconnect();
    }
    client.loop();

    unsigned long now = millis();
    // Only send values at a maximum frequency
    bool sendTime = now - lastSend > SEND_FREQUENCY;
    if (sendTime) {
        // convert to a string with 2 digits before the comma and 2 digits for precision
        dtostrf(kwh, 2, 4, kwhString);
        client.publish(mqtt_topic_kwh,kwhString);  // Publish kwh to MQTT topic
        lastSend = now;  // once every thing is published we update the send time
        // We calculate the power using the energy
        double watt = kwh * 1000.0 * 3600.0 / (double)SEND_FREQUENCY * 1000.0;
        dtostrf(watt, 4, 2, wattString);
        client.publish(mqtt_topic_watt,wattString);  // Publish watt to MQTT topic
        Serial.print("Watt:");
        Serial.println(wattString);
        // We calculate the accumulated energy since the begining using pulses count
        double kwhaccum = ((double)pulseCount/((double)PULSE_FACTOR));
        dtostrf(kwhaccum, 3, 3, kwhaccumString);
        client.publish(mqtt_topic_pulse,kwhaccumString);  // Publish pulses to MQTT topic
        kwh = 0;
    }

}

void onPulse()
{
    unsigned long newBlink = micros();
    unsigned long interval = newBlink-lastBlink;
    if (interval<10000L) { // Sometimes we get interrupt on RISING
            return;
    }
    // Every time there is a pulse, the energy consumption is 1 [pulse] / PULSE_FACTOR [pulses/kWh]
    // We also want to accumulate the energy (it will be initialized again once MQTT message is sent)
    kwh += 1.0 / (double)PULSE_FACTOR;
    lastBlink = newBlink;
    pulseCount++;
}

I’ve been using this setup for several days and I think it’s working nicely. These are the results in the HA frontend:

  • With TV switched on:

  • With TV switched off:

PS: Now I see that I need to change the icons.


#6

Hi, exelent project!
Can you share your final sensor setup on home assistant? (electricity.yaml)
I do not understand how you get the value “Accum energy”


#7

Hi @cosmok , thank you! I’m so sorry for the long silence; my raspi also takes holidays in August :slight_smile:
Maybe you already solved your questions but just in case…
This is my updated /home/homeassistant/.homeassistant/includes/sensors/electricity.yaml file:

- platform: mqtt
  name: "Inst power 01"
  state_topic: "ESP-energy-01/watt" 
  unit_of_measurement: "W"
- platform: mqtt
  name: "Inst energy 01"
  state_topic: "ESP-energy-01/kwh" 
  unit_of_measurement: "kWh"
- platform: mqtt
  name: "Accum energy 01"
  state_topic: "ESP-energy-01/pulse" 
  unit_of_measurement: "kWh"

and my updated /home/homeassistant/.homeassistant/includes/groups/electricity.yaml file:

electricity:
  name: 'Electricity'
  view: yes
  entities:
    - group.xtm18s_01

xtm18s_01:
  name: 'TV consumption'
  entities:
    - sensor.inst_power_01
    - sensor.inst_energy_01
    - sensor.accum_energy_01

Well, the accumulated energy (i.e. the total energy consumed by the electric device) can be easily calculated by knowing the pulses counted by the XTM18S. In the INO file it is done here (around line 150):

        // We calculate the accumulated energy since the begining using pulses count
        double kwhaccum = ((double)pulseCount/((double)PULSE_FACTOR));

Sorry again for the delayed response.


#8

Thanks Timseebeck.
I have mounted the system following your instructions but with an LCD counter XTM18SA and a SONOFF BASIC (it has power included and the port GPIO14 accessible)
Everything seems to work correctly


#9

Hi @timseebeck,
I’ve made a couple of changes to your code to make it connect with fixed IP to the router, send IP and MAC address via MQTT, and make the internal NodeMCU LED blink.

After a few hours the kWh counter returns to 0 even without NodeMCU being reset or powered off… What could be the problem?

You can see the code here: https://gist.github.com/jorgeassuncao/40956bbe4ac8db9fdb8f0526a4e0c0fe

Another question, how to modify the code to make this: Calculate energy price based on time of day and daylight savings time

Thank you very much for any help!


#10

Hi @j.assuncao ,

I’m not really an expert on this area so I have no clue about the behavior you mentioned with your NodeMCU. I’ve checked the differences between your code and mine and I don’t see why yours may lead to failure. Maybe you could try the following:
a) Increase the time between events sending: I have SEND_FREQUENCY = 120000 while yours is 15000, which maybe puts more stress to your MCU
b) Remove the part related to the LED blinking
c) Try on another MCU

While I’m not using this device right now, I have to say that I used it for 5 consecutive days without issues.

As for the other question, I’m going to answer in that thread.
Good luck!


#11

Hi @timseebeck,

I’ll be trying what you suggested this week. I suspect that the problem is the blinking LED. It stopped working right after i added the changes as you can see in this image.

What did you use to flash the NodeMCU, Arduino IDE or Platform .io? If you used Arduino what settings did you use for “flash mode” and “flash size”?

Thank you for your time.


#12

I used the Arduino IDE. My board selection was a “WeMos D1 R2 & mini” with CPU Frequency: “80 MHz”, Flash Size: “4M (3M SPIFFS)” and Upload Speed: “115200”. I don’t know anything about flash mode. Sorry.
You’re welcome. Good luck!


#13

Hi @timseebeck,

I removed the LED blinking part of the code and now is working fine, on the last 24h there was no reset. I’ll keep an eye on the next few days just to be sure. Thanks for the tip!

One more thing, is there a way to keep the accumulated kWh value between reboots/power-off of the device?


#14

I see two approaches here:
1.- Storing the value in the EEPROM of the microcontroller. I’m not 100% sure of what I’m going to say but it seems that the NodeMCU does not have an EEPROM. However, you could still use its flash memory. Be warned that too much writings/readings may wear the flash memory.
2.- Maybe the kWh value could be stored in HA and transferred to the NodeMCU somehow (just like it is done with the states of the relays).

In any case, I don’t have any clue about how any of the mentioned approaches could be implemented. And maybe there are other solutions. Let’s wait for someone smarter…

Offtopic: can you please tell me what you are using to make the plots? Is it Graphana? Thanks.


#15

Hi @timseebeck,

Tried EEPROM and SPIFFS methods today with no luck but i’m a total noob programming in Arduino… If anyone out there would like to help, just say something.

Using HA to store a variable is something that i don’t know how to do either. I’ll Google it and try on the next few days. Can we get someone smarter to help us!? Please!?

Offtopic: can you please tell me what you are using to make the plots? Is it Graphana? Thanks.

Yes, it’s InfluxDb server + Grafana addon on Hass.io.

Let’s wait for someone smarter. I really like you project and i’m planning to add a non-invasive current sensor to it.


#16

Hi again @timseebeck,

I think i’ve found the solution to keep the accumulated kWh value between reboots/power-off of the device!

The solution was in front of me all the time an i didn’t see it. Retaining the topic of kwhaccumString and then read it from broker after reboot/reset.


#17

That’s great!

So there was a third approach and it involves retaining the topic. Thanks for telling! Always learning from this community.


#18

Yes, and also solved the problem with the blinking LED. I’ll publish the code on Github soon and it will have a reference to you because it was based on you work.

You’re welcome! Sometimes we learn, sometimes we teach…

Thanks for sharing your work to!


#19

Hi Jorge.

I’m very interested in your code. When are you going to publish it on Github? :hugs:


#20

Hi @Pharizna. Sorry for the late reply…

You can get the code here. If you need any help or explanation just say!