Can someone help me extract complete lines {'XX': 'XXX'} and write to another file in linux? Help greatly appreciated

Rural_assist · November 7, 2019, 6:29pm

Hello!
I’ve currently got an ever expanding file in this format which is a constant print out from a serial port.
This is one line:

{'FW': '0308', 'SOC': '999', 'Relay': 'OFF', 'PID': '0x203', 'H10': '3', 'BMV': '700', 'TTG': '1692', 'H12': '0', 'H18': '690', 'I': '-6216', 'H11': '0', 'Alarm': 'OFF', 'CE': '-143', 'H17': '578', 'P': '-73', 'AR': '0', 'V': '11759', 'H8': '15632', 'H9': '93', 'H2': '-145', 'H3': '-62393', 'H1': '-68223', 'H6': '-478196', 'H7': '4', 'H4': '3', 'H5': '0'}

As I think the lines dont always get printed in complete sections. Hassio file sensor has problems setting sensors up on boot.

I need to read and print complete lines of this to a file, maybe once every 10 seconds. So HA has a chance to read and setup file sensors.

Can anyone supply me with a script that can do this?

The current serial printed script resides at:


/usr/share/hassio/homeassistant/vedirect-bmv/bmv700.txt

The new parsed file can also print to this folder.

Any help greatly appreciated!

dap35 · November 7, 2019, 7:12pm

I think what you need is something like this :

tail -2 bmv700.txt  | head -1 > stuff-I-want-to-parse-for-HA.txt

This will take the next to last line in the file and put that into a file you can parse for HA. If you want to be extra safe, make it tail -3.

Rural_assist · November 7, 2019, 7:28pm

Amazing! I ran (in terminal):

tail -2 bmv700.txt  | head -1 > stuff-I-want-to-parse-for-HA.txt

It created a file with just one line in it! Great stuff.

Excuse my lack of knowledge… How do I make a python script (or other) to repeat that command every 10 secs? I can then pop that job into crontab to start on reboot

dap35 · November 7, 2019, 7:46pm

I would just create a bash script on your Linux host (myscipt.sh) , change it to executable (chmod +x myscript.sh) . Additionally, I would not write the bmv700.txt file out to the pi running HA, since it runs across the network and also will increase the wear on the SD card, leading to premature aging .

#/bin/sh
tail -2 bmv700.txt  | head -1 > /path-to-HA-mount/config/stuff-I-want-to-parse-for-HA.txt

Rural_assist · November 7, 2019, 9:00pm

Thanks dap35!
I appreciate you taking the time to help

I have done several NUC reboots, followed by several stand alone hassio reboots and the sensors have set up perfectly every time (rather than 20 reboots till they all show!)
Your script worked!

Popping some info on here to help others…

So made I made the script loop:

My filename: bmv700.sh

#/bin/sh
while [ true ]
do
    tail -3 bmv700.txt  | head -1 > /usr/share/hassio/homeassistant/vedirect-bmv/bmv700Parsed.txt
    sleep 10
done

Its now writing the new parsed file every 10 secs.

To get the script to run at startup/reboot:
In terminal run:

crontab -e

Edit the file to add:

@reboot cd /usr/share/hassio/homeassistant/vedirect-bmv && bash bmv700.sh

Very pleased. Thanks again dap35

dap35 · November 8, 2019, 3:00am

I’m glad that helped.

dap35 · November 9, 2019, 10:17pm

If you use mqtt for anything else in your environment, I played around today and figured out enough of a python script to parse the file and publish the individual readings to an MQTT server.

SteveDinn · November 9, 2019, 11:10pm

Rather than have a bash script and write it to a file that is later parsed, you could make this a command line sensor:

sensor:
  - platform: command_line
    name: bmv700
    command: tail -2 bmv700.txt  | head -1
    scan_interval: 10

Rural_assist · November 9, 2019, 11:22pm

dap35 That would be very useful!
Ive just set up an ESP home node (wifi). Still havent worked with MQTT yet… Thats the next learning curve.

SteveDinn I’ll do an experiment with that and see how well it works! Dap35’s solution has been working flawlessly so far.
Suppose if it does. It cuts out the middle process achieving the same result

Rural_assist · November 9, 2019, 11:54pm

So

  - platform: command_line
    name: Avg Discharge
    scan_interval: 10
    command: 'tail -2 /config/vedirect-bmv/bmv700.txt  | head -1'
    value_template: >
      {{ '%0.1f'|format((value | regex_findall_index("'H3': '(-*\d+)'") | int) / 1000) }}
    unit_of_measurement: 'Ah'

Appears to achieving the same result so far (will take a few reboots to see if its stable).

I am presuming the initial errors were HA seeing incomplete lines of data on boot. Hence giving errors on the sensors.

I had already setup a command line sensor with this config:

  - platform: command_line
    name: Cumulative AH
    scan_interval: 10 
    command: 'tail -n 1 /config/vedirect-bmv/bmv700.txt'
    value_template: >
      {{ '%0.1f'|format((value | regex_findall_index("'H6': '(-*\d+)'") | int) / 1000) }}
    unit_of_measurement: 'Ah'

But the sensors were sporadic on working!

Definitely picking up new knowledge with the help of you guys.
Thanks

dap35 · November 10, 2019, 1:42am

Basically, a minor tweek to the bash script to create a json file which is kept on your linux host.

**# the sed command is used to replace the single quotes with double quotes and make this a valid json file.**

`tail -2 bmv700.txt | head -1 | sed s/\'/\"/g > parse2.txt`

Also run on the linux host, this will read the json file and publish the sensor readings up to the mqtt server.

import json
import paho.mqtt.client as mqtt
with open('parse2.txt', 'r') as sensordata:
data=sensordata.read()
# parse file
obj = json.loads(data)
#setup and connect to MQTT server
broker_url = “my-mqtt.server.local”
broker_port = 1883
mq_user = “user_namer”
mq_pw = “passwordf"
#
#mqtt topic to publish to
#
topic2 = "test/H2"
topic3 = "test/H3"
topic5 = "test/H5"
#
client = mqtt.Client()
client.username_pw_set(mq_user, password=mq_pw)
client.connect(broker_url, broker_port)
#
client.publish(topic=topic2, payload=int(obj['H2']), qos=0, retain=False)
client.publish(topic=topic3, payload=int(obj['H3']), qos=0, retain=False)
client.publish(topic=topic5, payload=int(obj['H5']), qos=0, retain=False)

Its crude, and doesn’t loop, but it could be expanded, and set up as a service.

Rural_assist · November 10, 2019, 10:04am

Will get cracking on setting up that MQTT server!
Looking forward to getting this working.

Thanks dap35

dap35 · November 12, 2019, 8:44pm

Here is a cleaner option, in two parts. I believe this captures the readings you were looking for.

grab.sh:==>

#!/bin/sh
tail -3 bmv700.txt | head -1 | sed s/\'/\"/g > parse2.txt

python code to run as a service:==>

#!/usr/bin/python3
import json
import paho.mqtt.client as mqtt
import time
import os
broker_url = "mqtt.local.int"
broker_port = 1883
delay = 10
mq_client = "bmv700host"
mq_user = "username"
mq_pw = "password"
topic1 = "bmv700/AVG_Discharge"
topic2 = "bmv700/CUM_AH"
status = "bmv700/connection_status"
lwm = "offline"
#
client = mqtt.Client(mq_client)
client.will_set(status, lwm, qos=1, retain=True)
client.username_pw_set(mq_user, password=mq_pw)
client.connect(broker_url, broker_port)
client.publish(topic=status, payload="online", qos=0, retain=False)
#
client.loop_start()
rc=1
while rc > 0:
        #### Edit path to grab.sh to match your environment ####
        os.popen('/home/pi/grab.sh')
        with open('parse2.txt', 'r') as sensordata:
            data=sensordata.read()
        obj = json.loads(data)
        H3 = int(obj['H3'])
        H6 = int(obj['H6'])
        client.publish(topic=topic1, payload=H3, qos=0, retain=False)
        client.publish(topic=topic2, payload=H6, qos=0, retain=False)
        print("Parsing data", H3, H6)
        time.sleep(delay)

Rural_assist · April 28, 2020, 2:31pm

Got round to your script, it worked a treat!
Much preferring using MQTT.

Thanks Dap