There is a similar thing here that I’m using on two remote GPIO to MQTT bridges at the moment:
It does not have network throughput, but it does have discovery.
I really like such a srcipt but get errors running in on thr rpi:
Traceback (most recent call last):
File “rpi-cpu2mqtt.py”, line 71, in
cpu_load = check_cpu_load()
File “rpi-cpu2mqtt.py”, line 31, in check_cpu_load
cpu_load = p.split(“average:”)[1].split(",")[0].replace(’ ', ‘’)
TypeError: a bytes-like object is required, not ‘str’
I am new in Python. Any suggestion?
what version of python do you use - I have tested and developed this on python 2 as the mqtt library was not available for version 3 when I wrote this couple of years ago. If you are on python 3 pls try version 2.
I plan to test with python 3 and make corrections if needed when I have time - but now I plan to first fix the cpu load % and implement the network traffic (also I have plans to add to the configuration ways to choose what is monitored)
This script looks good and the graphs are better than mine - sorry for duplicating I was not aware of this and it was not available when I did mine.
You have nothing to apologise for.
Options are always good.
The graphs are not part of it, that’s just Lovelace (and maybe some custom cards).
1 Like
Instead of defining the sensors in YAML, you can use the following script to define them via MQTT Discovery. The advantage is that all of the sensors will be associated with a device (that’s the one thing you currently can’t do in YAML is define MQTT Sensors with a device). It’s a handy way to organize the sensors (they are all associated with a specific device) and also permits the creation of device-oriented automations.
Modify the following script to suit your environment’s requirements:
#script:
create_rpi4_sensors:
alias: "Create RPI4 sensors via MQTT Discovery"
sequence:
- service: mqtt.publish
data:
topic: homeassistant/sensor/rpi4_cpuload/config
retain: true
payload: >
{
"name": "RPI4 CPU Load",
"unit_of_measurement": "%",
"state_topic": "masoko/rpi4/cpuload",
"unique_id": "rpi4_cpu_load",
"device": {
"identifiers": ["rpi4_status_monitor"],
"name": "Raspberry Pi",
"model": "RPI4",
"manufacturer": "Raspberry Pi Foundation",
"sw_version": "X.XX"
}
}
- service: mqtt.publish
data:
topic: homeassistant/sensor/rpi4_cputemp/config
retain: true
payload: >
{
"name": "RPI4 CPU Temperature",
"unit_of_measurement": "°C",
"state_topic": "masoko/rpi4/cputemp",
"unique_id": "rpi4_cpu_temp",
"device": {
"identifiers": ["rpi4_status_monitor"],
"name": "Raspberry Pi",
"model": "RPI4",
"manufacturer": "Raspberry Pi Foundation",
"sw_version": "X.XX"
}
}
- service: mqtt.publish
data:
topic: homeassistant/sensor/rpi4_diskusage/config
retain: true
payload: >
{
"name": "RPI4 Disk Usage",
"unit_of_measurement": "%",
"state_topic": "masoko/rpi4/diskusage",
"unique_id": "rpi4_disk_usage",
"device": {
"identifiers": ["rpi4_status_monitor"],
"name": "Raspberry Pi",
"model": "RPI4",
"manufacturer": "Raspberry Pi Foundation",
"sw_version": "X.XX"
}
}
- service: mqtt.publish
data:
topic: homeassistant/sensor/rpi4_voltage/config
retain: true
payload: >
{
"name": "RPI4 Voltage",
"unit_of_measurement": "V",
"state_topic": "masoko/rpi4/voltage",
"unique_id": "rpi4_voltage",
"device": {
"identifiers": ["rpi4_status_monitor"],
"name": "Raspberry Pi",
"model": "RPI4",
"manufacturer": "Raspberry Pi Foundation",
"sw_version": "X.XX"
}
}
- service: mqtt.publish
data:
topic: homeassistant/sensor/rpi4_clockspeed/config
retain: true
payload: >
{
"name": "RPI4 System Clock Speed",
"unit_of_measurement": "hz",
"state_topic": "masoko/rpi4/sys_clock_speed",
"unique_id": "rpi4_sys_clock_speed",
"device": {
"identifiers": ["rpi4_status_monitor"],
"name": "Raspberry Pi",
"model": "RPI4",
"manufacturer": "Raspberry Pi Foundation",
"sw_version": "X.XX"
}
}
3 Likes
Thanks I’ll look at that looks really useful and interesting. I’ll look at it later.
Now I am updating the script as I got inspired by all the nice comments here - I just fixed the CPU load so its really in %
Memory and swap are implemented - both in % used
2 Likes
Thank you very much for your fast and helpful response, that works! Great script: easy and very useful!
I did something similar, but I needed different informations.
My goal was to publish two tasmota-like topics every x minutes with STATE and SENSOR jsons.
tele/p2p_server/LWT Online
tele/p2p_server/STATE {"Time":"2020-04-29T18:53:15","Uptime":"54T21:04:37","CPULoad":2.7,"MemoryTotal":926.1,"MemoryAvailable":512.4,"MemoryPercent":44.7,"DiskTotal":77.8,"DiskFree":48.7,"DiskPercent":34.0,"POWER":"ON"}
tele/p2p_server/SENSOR {"Time":"2020-04-29T18:53:15","AMULE":{"Upload":25.01,"Download":0.0},"TRANSMISSION":{"Upload":0.0,"Download":0.0}}
My script once started will loop forever and publishes an LWT topic to share the availability. You can easily convert it into a service and manage it with systemctl command.
Here is the code I used:
#!/usr/bin/python3
import os
import re
import time
from datetime import datetime
import paho.mqtt.client as mqtt
import psutil
def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
client.subscribe("tele/p2p_server/#")
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
def time_now():
data_now_obj = datetime.now()
data_now_str = data_now_obj.strftime("%Y-%m-%dT%H:%M:%S")
return (data_now_str.strip())
def uptime():
data_now_obj = datetime.now()
data_now_str = data_now_obj.strftime("%Y-%m-%dT%H:%M:%S")
data_uptime_str = os.popen("uptime -s").readline().strip()
data_uptime_obj = datetime.strptime(data_uptime_str,"%Y-%m-%d %H:%M:%S")
uptime_obj = data_now_obj - data_uptime_obj
uptime_str = "%dT%02d:%02d:%02d" % (uptime_obj.days, (uptime_obj.seconds // 3600), (uptime_obj.seconds % 3600 // 60), (uptime_obj.seconds % 60))
return (uptime_str.strip())
def load():
cpu = psutil.cpu_percent(interval=None)
mem = psutil.virtual_memory()
disk = psutil.disk_usage('/mnt/usbdisk')
temp = "\"CPULoad\":%s,\"MemoryTotal\":%s,\"MemoryAvailable\":%s,\"MemoryPercent\":%s,\"DiskTotal\":%s,\"DiskFree\":%s,\"DiskPercent\":%s" % (cpu,(round(mem.total / 1048576,1)),(round(mem.available / 1048576,1)),mem.percent,(round(disk.total / 1073741824,1)),(round(disk.free / 1073741824,1)),disk.percent)
return (temp)
def transmission():
temp = os.popen("transmission-remote -n transmission:frigor -l|grep ^Sum:").readline()
return (re.sub(r'^.*(\d+\.\d+)\s+(\d+\.\d+)$', r'"TRANSMISSION":{"Upload":\1,"Download":\2}', temp.strip()))
def amule():
dlstr = 0
upstr = 0
temp = os.popen("amulecmd -P frigor -p 4713 -c 'status'").read()
for item in temp.split("\n"):
if "Download" in item:
dlstr = re.sub(r'^.+\s(\d+\S*).*$', r'\1', item.strip())
if "bytes" in item:
dlstr = round(eval('float(dlstr)/1024'),1)
elif "Upload" in item:
upstr = re.sub(r'^.+\s(\d+\S*).*$', r'\1', item.strip())
if "bytes" in item:
upstr = round(eval('float(upstr)/1024'),1)
temp = "\"AMULE\":{\"Upload\":%s,\"Download\":%s}" % (upstr, dlstr)
return (temp)
qos=0
retain=True
topic_id="YOUR_TOPIC"
client=mqtt.Client(topic_id)
client.username_pw_set("USERNAME", password="SECRET_PWD")
topic = "tele/%s/LWT" % topic_id
payload = "Offline"
client.will_set(topic, payload, qos, retain)
client.reconnect_delay_set(min_delay=1, max_delay=120)
client.connect("IP_ADDRESS", 1883, 60)
payload = "Online"
client.publish(topic, payload, qos, retain)
#client.loop_forever()
while True:
retain=False
topic = "tele/%s/STATE" % topic_id
payload = "{\"Time\":\"%s\",\"Uptime\":\"%s\",%s,\"POWER\":\"ON\"}" % (time_now(),uptime(),load())
client.publish(topic, payload, qos, retain)
topic = "tele/%s/SENSOR" % topic_id
payload = "{\"Time\":\"%s\",%s,%s}" % (time_now(),amule(),transmission())
client.publish(topic, payload, qos, retain)
time.sleep(60)
I am not a big fan of looping scripts that stay in the memory all the time and if they hung for some reason they can’t recover by them-self. In its early stages my script was also a looping one but then I found its much more stable and controllable if I just run it with a cronjob when I need it.
I also tried to use as less dependencies as possible - most of my checks are done with bash commands and the only additional python module I use it the paho.mqtt.client.
This is just a personal opinion based on my experience maybe there are advantages in using a looping script and not a cron or using python modules than bash commands. I’ll be happy if some body with more experience and deeper python understanding can comment on this.
In my (limited) experience I noticed that multiple accesses to mqtt server were taking more time and resources than just waiting for the next publish time…
Me too
I was also thinking about this and plan to test it - my mqtt server which is hosted on rpi 0 is getting around 40-45 messages per min and is getting overloaded.Plan to add an option to send all data as a group message and if this will reduce the load on the mqtt server.
I have updated the script to send one json message but can’t read the separate values - the payload example is in the post description and I believe this is the error - thus my template is different - but I don’t have a sensor with template matching the error:
Log Details (ERROR)
Logger: homeassistant.helpers.template
Source: helpers/template.py:284
First occurred: 4:33:43 PM (78 occurrences)
Last logged: 4:53:21 PM
Error parsing value: 'value_json' is undefined (value: None, template: {{ value_json.status == 'enabled' }})
If someone can help with this or suggest payload format easier to parse in hass
thanks
UPDATE: I made it with CSV - will publish it shortly - this way even the message is shorter
If you publish the payload like this:
{"used_space": 25, "sys_clock_speed": 1500, "cpu_temp": 43.0, "voltage": 0.8500, "cpu_load": 1.25, "memory": "False", "swap": "False"}
then an MQTT Sensor configured like this:
- platform: mqtt
name: "pi monitor"
state_topic: "test"
value_template: "{{ value_json.cpu_load }}"
json_attributes_topic: "test"
json_attributes_template: >
{ "used_space": {{value_json.used_space}},
"sys_clock_speed": {{value_json.sys_clock_speed}},
"cpu_temp": {{value_json.cpu_temp}},
"voltage": {{value_json.voltage}},
"memory": "{{value_json.memory}}",
"swap": "{{value_json.swap}}" }
will result in this:
1 Like
Thanks @123 I already made it with CSV message - this way the message looks unreadable but is smaller and the main goal for me was to reduce the load on the MQTT broker so I’m happy with the result.
I’ll refer to your comment next time I have problems with parsing JSON in Hass.
To be honest I am still not sure what was wrong with my setup - as my template looks like yours - only using single brackets instead of double - can this be the issue?
The new version of my script is published on github for anybody interested - I still don’t know if this will reduce the load on my MQTT broker - I also plan to put a random delay before sending the message so I can de-synchronize the messages - now all my 8 raspberries shoot at the MQTT broker at the exact same second and its not cool for him.
What kind of sensor are you using in Home Assistant to read and extract data from the CSV?
The mosquito MQTT Broker is efficient. Check the CPU load of the machine hosting mosquitto to see if “loading” is as excessive as you think.
I am using mqtt sensors like this:
- platform: mqtt
name: 'rpi4 cpu load'
state_topic: 'masoko/rpi4'
value_template: '{{ value.split(",")[0] }}'
unit_of_measurement: "%"
I have checked the load on the machine hosting the mqtt broker and it reaches 100% when all 8 rpis start sending each 7 messages a the same moment - that is why I grouped the messages in one csv message.
My mqtt broker is a rpi 0w - its not a very powerful board and these 56 messages send at once were giving it hard time + there are other messages from temperature and humidity sensors and motion sensors
My biggest problem was that the automatons triggered by motion were delayed. The screenshot above is not with all 56 messages - I have disabled some that I don’t need. Maybe my mqtt broker is not configured or optimized properly - I am using a default installation but it works.
The JSON payload’s few extra bytes, versus CSV, isn’t the issue. The RP0 simply isn’t the optimal platform for the level of ‘burst traffic’ you’re generating.
Plus, the CSV format eliminates the possibility of using Home Assistant’s native ability to parse JSON. Your original payload wasn’t valid JSON because it employed single-quotes.
Invalid:
Valid:
Thank you for quick response and explanation.
I composed it with python json module and validated with http://jsonviewer.stack.hu/ all seemed OK and I had no doubts in the json formatting.
I know the rpi0 is not the best hardware but I think I’ll still make it work with combining the messages in one and de-synchronizing the traffic so its not burst.
I’ll go with the CSV for now as I have reconfigured all my sensors in hass - I have to try the auto discovery method described by you above - it looks faster than updating my sensors.yaml manually.
Thanks again.