Sensor Handling with Python

Development
1

Calling all Python experts!

I am working with some sensors that connect to hub. The sensors send their data to the hub in intervals and it looks like this:

#[device address],[distance],[time],[reading counter]

#1,3200.000,18/06/02 8:00:05,271
#2,3200.000,18/06/02 8:05:05,271
#3,3200.000,18/06/02 8:10:05,271

Executing this python code returns the expected value above in stdout.

import serial

ser =serial.Serial ('/dev/ttyUSB0', 9600)
response = ser.read(35)

print response

I would like to come up with a better way to handle these readings. For example if the response string starts with #1 then write the [distance] portion of that string to a file named sensor 1.txt

import serial

ser = serial.Serial('/dev/ttyUSB0', 9600) #listen on USB0 @ 9600 buad
response = ser.read(35) # read 35 characters of response from device connected to USB0
# The following liines need to send readings to specific files based on the incoming sensor data...
if response.startswith('#1')
    file = open("sensor_1.txt", "w")	
	file.write(response)									# I would also like to just print the distance reading so string.str #1,        ,18/06/02 8:01:05,271,184
	file.close														# the empty spaces would be the distance reading that is written to the file.
elif response.startswith('#2')
    file = open("sensor_2.txt","w")
	file.write(response)
	file.close
elif response.startswith('#3')
    file = open("sensor_3.txt","w")
	file.write(response)
	file.close
else
    file = open("sensor_error","w")
	file.write('Check Sensor Connection')
	file.close
endif:

My thought is that if I get the correct chunk of data to the correct file then I can use file sensors for each device. I would also like it if this would continue to run or loop. As of right now it exits once the response from the sensor is received.

If you couldn’t tell I have no coding experience but have been using HA since ~ 0.12. I have a good grasp on templates and other components.

If anyone has a better ideas I am all ears!

Regards,

2

A couple of things I can see.

To loop through the input, you need to replace

ser = serial.Serial('/dev/ttyUSB0', 9600) #listen on USB0 @ 9600 buad

with

with serial.Serial('/dev/ttyUSB0', 9600) as ser:

and then indent the rest of the file. See https://pyserial.readthedocs.io/en/latest/shortintro.html#short-introduction

Always reading 35 characters assumes that your serial port is perfect. They never are, and sometimes you will get a missed character. This will then put your line reading out by one, and you will have to restart the program to get it back in alignment. So it is better to do

response = ser.readline()

To read up until the next newline. You may have to do some additional fiddling if you don’t have a traditional newline character, but it is normally ok by default. You then need to check that the input is indeed 35 characters long (and checking any other criteria would be good too).

A minor point is that you could open all the output files before you start the loop, rather than for each new output, but this is just for efficiency.

And personally, I would send MQTT messages to HA, rather than using a file, but how easy that is depends on whether you already have MQTT set up.

3

@gpbenton

Thanks for the input! I believe you have me on the right foot now.

Here is my new code:

import serial
import string

def main():

    with serial.Serial ('/dev/ttyUSB0', 9600) as ser:  # listen on USB0 @ 9600 baud
        response = ser.readline() 
        str = response
if str.startswith('#1'):
    file = open("sensor_1.txt", "a")	
	file.write(response)									# I would also like to just print the distance reading so string.str #1,        ,18/06/02 8:01:05,271,184
	file.close														# the empty spaces would be the distance reading that is written to the file.
elif str.startswith('#2'):
    file = open("sensor_2.txt","a")
	file.write(response)
	file.close
elif str.startswith('#3'):
    file = open("sensor_3.txt","a")
	file.write(response)
	file.close
else
    file = open("sensor_error.txt","a")
	file.write('Check Sensor Connection')
	file.close
	
main()

The sensor readings are now being appended to their respected files.

Now I need this to:
1: “loop” or continue to listen and write after the first response. As of right now I type $ python sensor.py into the terminal and it waits for the response. I then have to execute the script again.

2: I need to write a truncated version of the output to the file.
Current Output: #1,3200.000,18/06/02 8:00:05,271
Needed Output: Everything in between the first and second commas: 3200.00

your thoughts are greatly appreciated!!

4

If you follow my initial post and indent the rest of the file to match str = response, you will achieve this.
Edit: Except the main() line of course :roll_eyes:

Try

output = response.split(',')[1]
5

I wanted to check out the MQTT discovery, so I decided to use this as an example. This should create some sensors and then send an MQTT message to a different sensor for each line, depending on the starting number.

import serial
import paho.mqtt.publish as mqtt
import time

# Messages to configure sensors by MQTT discovery
config = [
        {"topic":"homeassistant/sensor/one/config", "payload":'{"name":"sensor one"}'},
        {"topic":"homeassistant/sensor/two/config", "payload":'{"name":"sensor two"}'},
        {"topic":"homeassistant/sensor/three/config", "payload":'{"name":"sensor three"}'}
        ]

mqtt.multiple(config)

# a little delay to allow HA to process the config message
time.sleep(1)

with serial.Serial ('/dev/ttyUSB0', 9600) as ser:  # listen on USB0 @ 9600 baud
    while True:
        response = ser.readline()
        if len(response) != 33:
            mqtt.single("homeassistant/sensor/error/state", payload="Sensor Error")
            continue
        try:
            data = response.split(',')[1]
        except IndexError:
            mqtt.single("homeassistant/sensor/error/state", payload="Sensor Error")
            continue

        if response.startswith('#1'):
            mqtt.single("homeassistant/sensor/one/state", payload=data)
        elif response.startswith('#2'):
            mqtt.single("homeassistant/sensor/two/state", payload=data)
        elif response.startswith('#3'):
            mqtt.single("homeassistant/sensor/three/state", payload=data)
        else:
            mqtt.single("homeassistant/error", payload="Sensor Error")
            continue

Edit: Added missing while loop

2 Likes
6

@gpbenton

Sorry for the delay and thank you so much for helping me with this!! I am learning a great deal!!

I am going to review and test everything then get back to you!!

Highest Regards,

7

@gpbenton Here is what I have working thus far. Let me know if I can improve anything. I also dont know if the exceptions I called are correct or if it matters. Once again this is my first crack at Python so I am grateful for your assistance!

I plan on trying MQTT for this because it would be a little cleaner.

import serial
import math
import string

from decimal import getcontext, Decimal
getcontext().prec = 4

with serial.Serial ('/dev/ttyUSB0', 9600) as ser:  # listen on USB0 @ 9600 baud
    while True:
        response = ser.readline()
        if len(response) >= 40:
            file = open('sensor_error.txt', 'a')
            file.write('\n')
            file.write("Reply Is Too Long")
            file.close
            continue
        try:
            data = response.split(',')[1]    # Get the needed data from string
        except IndexError:
            file = open('sensor_error.txt','a')
            file.write('\n')
            file.write("Data Not Properly Truncated")
            file.close
            continue
        if data.startswith("E"):    # Write Error Codes to file
            file = open('sensor_error.txt', 'a')
            file.write('\n')
            file.write(response + "Look up Error Code")
            file.close
        elif data.startswith("V"):    # When starting the script the first reading is always V3.5 so I call it the "initializing"
            file = open ('sensor_error.txt', 'a')
            file.write('\n')
            file.write("System Initializing")
            file.close
            continue
        try:
            data = float(data)    # Make string a float
        except IndexError:
            file = open('sensor_error.txt', 'a')
            file.write('\n')
            file.write("Data Not Correctly Formatted")
            file.close
            continue
        try:
            value = Decimal(data) * Decimal(3.2808399)    # Convert to feet
        except IndexError:
            file = open('sensor_error.txt', 'a')
            file.write('\n')
            file.write("Value Not Converted to Feet")
            continue
        if response.startswith("#1"):    # Write sensor measurements to a file
            file = open('sensor_1.txt', 'a')
            file.write('\n')
            file.write(str(value))
            file.close
        elif response.startswith("#2"):
            file = open('sensor_2.txt', 'a')
            file.write('\n')
            file.write(str(value))
            file.close
        elif response.startswith("#3"):
            file = open('sensor_3.txt','a')
            file.write('\n')
            file.write(str(value))
            file.close
        else:
            file = open('sensor_error.txt','a')    # Catch all error
            file.write('\n')
            file.write("Check Sensor Connection")
            file.close
            continue
8

Looks fine afaics, except as you thought, the conversions raise different exceptions

should be ValueError, as this is what is raised when python cannot convert the string to a number. You can check these things out using the command line like

graham@naos-kde:~
$ python3
Python 3.6.5 (default, Apr  1 2018, 05:46:30) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> float('a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: could not convert string to float: 'a'
>>>
9

Thank you!

I am now trying it with MQTT.

When the script is initiated it makes the sensors in HA.

I then receive this error:

Tue May 29 2018 09:46:42 GMT-0500 (Central Daylight Time)

BrokerProtocolHandler Unhandled exception in reader coro: IncompleteReadError('0 bytes read on a total of 1 expected bytes',)

I also receive it every time a measurement comes in from the physical sensor.

I am just using the default embedded broker with discovery.

Any thoughts?

10

With the HA embedded broker I believe you need to specify that the client must use the 3.1.1 version of the protocol, which is set using the protocol parameter, see https://www.eclipse.org/paho/clients/python/docs/#single

If you have a password set up for HA, you will also need to send that to connect with the embedded broker (from memory, the username is homeassistant).

11

Would I just pass that with my mqtt.single("homeassistant/sensor/one/state", payload=reading, protocol=mqtt.MQTTv311) ?

This is what I tried anyway and got error AttributeError: 'module' object has no attribute 'MQTTv311'

I am getting all the correct data into HA but it raises the error every time.

12

I expect that where you copied the protocol parameter from defines mqtt differently to your script.

13

Ok I believe I understand what you mean and have changed the code. The script is working again with out erring out. With that said I do continue to get the error entries in HA.

I might try installing Mosquitto broker unless you have any other ideas.

Feel free to tap out if you want lol. You have helped me out so much! Thanks again!

Wed May 30 2018 08:24:39 GMT-0500 (Central Daylight Time)

BrokerProtocolHandler Unhandled exception in reader coro: IncompleteReadError('0 bytes read on a total of 1 expected bytes',)


import serial
import math
import string
import paho.mqtt.publish as publish
import paho.mqtt.client as mqtt
import time

from decimal import getcontext, Decimal
getcontext().prec = 4

# Messages to configure sensors by MQTT discovery
config = [
        {"topic":"homeassistant/sensor/one/config", "payload":'{"name":"Laser 1M", "unit_of_measurement":"feet"}'},
        {"topic":"homeassistant/sensor/two/config", "payload":'{"name":"Laser 2M", "unit_of_measurement":"feet"}'}
        ]

publish.multiple(config, protocol=mqtt.MQTTv311)

time.sleep(1) 

with serial.Serial ('/dev/Laser', 9600) as ser:  # listen on USB0 @ 9600 baud
    while True:
        response = ser.readline()
        if len(response) >= 40:
            file = open('sensor_error.txt', 'a')
            file.write('\n')
            file.write(response + "Reply Is Too Long")
            file.close
            continue
        try:
            data = response.split(',')[1]
        except IndexError:
            file = open('sensor_error.txt','a')
            file.write('\n')
            file.write("Data Not Properly Truncated")
            file.close
            continue

        if data.startswith("E"):
            file = open('sensor_error.txt', 'a')
            file.write('\n')
            file.write(response + "Look up Error Code")
            file.close
        elif data.startswith("V"):
            file = open ('sensor_error.txt', 'a')
            file.write('\n')
            file.write("System Initializing")
            file.close
            continue
        try:
            data = float(data)
        except ValueError:
            file = open('sensor_error.txt', 'a')
            file.write('\n')
            file.write("Data Not Correctly Formatted")
            file.close
            continue
        try:
            value = Decimal(data) * Decimal(3.2808399)
        except ValueError:
            file = open('sensor_error.txt', 'a')
            file.write('\n')
            file.write("Value Not Converted to Feet")
            file.close
            continue
        try:
            reading = str(value)
        except ValueError:
            file = open('sensor_error.txt', 'a')
            file.write('\n')
            file.write("Value Could Not Be Returned to String")
            file.close
            continue

        if response.startswith("#1"):
            publish.single("homeassistant/sensor/one/state", payload=reading, protocol=mqtt.MQTTv311)
        elif response.startswith("#2"):
            publish.single("homeassistant/sensor/two/state", payload=reading, protocol=mqtt.MQTTv311)
        elif response.startswith("#3"):
            file = open('sensor_3.txt','a')
            file.write('\n')
            file.write(reading)
            file.close
        else:
            file = open('sensor_error.txt','a')
            file.write('\n')
            file.write("Check Sensor Connection")
            file.close
            continue
14

I couldn’t see anything wrong in the script, so I set up a test configuration using the embedded broker, and I still don’t see any errors, even when using the default protocol version from my original script, and doing the conversions you are attempting.

This is my mqtt configuration

mqtt:
    discovery: true

so I suspect now that the problem may be in the payload you are sending.

Try printing out the payload just before you send it, to check it is something valid, and perhaps try a simpler script just sending some strings as a payload, to check everything else is set up.

15

I will dig in!

Thanks!

16

Man I am at a loss here. Just running the script below causes the error.

import paho.mqtt.publish as publish
import paho.mqtt.client as mqtt

from decimal import getcontext, Decimal
getcontext().prec = 4

# Messages to configure sensors by MQTT discovery
config = [
        {"topic":"homeassistant/sensor/one/config", "payload":'{"name":"Laser 1M", "unit_of_measurement":"feet"}'},
        {"topic":"homeassistant/sensor/two/config", "payload":'{"name":"Laser 2M", "unit_of_measurement":"feet"}'}
        ]

publish.multiple(config, protocol=mqtt.MQTTv311)

I am running HA 0.70.0 on a Pi3 in a python venv.

Here is the paho info

pi@proto-gateway:~ $ pip show paho-mqtt
Name: paho-mqtt
Version: 1.3.1
Summary: MQTT version 3.1.1 client class
Home-page: http://eclipse.org/paho
Author: Roger Light
Author-email: [email protected]
License: Eclipse Public License v1.0 / Eclipse Distribution License v1.0
Location: /usr/local/lib/python2.7/dist-packages
Requires:

My configuration.yaml

mqtt:
  discovery: true
17

I just ran this script on my test configuration and it worked fine.

The only difference I can see is that I am still on HA version 0.68.1, but I can’t imagine MQTT has been broken in later releases. The only thing I can suggest atm is installing mosquitto as the broker.

18

I will give that a shot!

19

I just tried your script against 0.70.0, and I do get the error message you report, so it looks like it was introduced since 0.68.1.

However, despite the warning, the sensors are created successfully, so if you haven’t started mosquitto yet, I would just ignore the warning.

20

I have not started yet. I suppose I can suppress those by adding additional parameters to logger:.

Thanks for all your effort on this! Things are working great and it was a lot of fun learning a bit more about Python

On a side note do you know anything about AWSIoT?

Or have you done anything with MQTT state stream?

I will probably create a new topic for my next venture lol