Run python script when HA starts

Configuration
13

Some feedback on various trials of shell_command:

without the dot prefix to the path, I get error code 2 - file not found? Not clear…

with the dot prefix to the path I get this in HA’s log:

2023-03-28 11:35:26.920 ERROR (MainThread) [homeassistant.components.shell_command] Timed out running command: python ./modbus/modbus.py, after: 60s
Traceback (most recent call last):
File “/usr/local/lib/python3.10/asyncio/streams.py”, line 502, in _wait_for_data
await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/lib/python3.10/asyncio/subprocess.py”, line 195, in communicate
stdin, stdout, stderr = await tasks.gather(stdin, stdout, stderr)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/lib/python3.10/asyncio/tasks.py”, line 456, in wait_for
return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/src/homeassistant/homeassistant/components/shell_command/init.py”, line 83, in async_service_handler
stdout_data, stderr_data = await asyncio.wait_for(
File “/usr/local/lib/python3.10/asyncio/tasks.py”, line 458, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

Using ps in the terminal doesn’t show modbus.py is running.

Conclusion: neither /path (no dot prefix) or ./path (with dot prefix) work…

14

Have you ever looked at AppDaemon?

I initially installed it to run the “schedy” heating scheduler but I ended up doing a lot of things with it

  • It plays well with home assistant and/or mqtt
  • When home assistant start / restart AppDaemon restart all its applets/script so your usecase above is easy as pie

https://appdaemon.readthedocs.io/en/latest/

15

That’s quite literally the killer. Apparently shell_command terminates any process after 60s. Full stop. No questions. No prisoners. Another dead duck floating in the HA pool, another wasted morning I shall never get back.

One of the main if not the main problem with HA is the failure of basic documentation. Why doesn’t the Shell_command documentation say the command WILL be terminated after 60s? All it mentions is a passing throwaway “In case a command results in a non 0 exit code or is terminated after a timeout of 60 seconds, the result is logged to Home Assistant log.” Nothing about the termination being by design.

I may give the pyscripts a go once I have got over my frustration.

@jfparis - I did look at AppDaemon but found the documentation was written in gobbledygook and gave up. I may have to have another look if pyscripts fails.

16

Yes. Shared that feeling. I worked my way through the examples.

17

I believe the operative word is timeout in “timeout of 60 seconds”.

Perhaps it’s not as clear as you want it to be but it also wasn’t crystal clear that your python script apparently runs forever:

The script collects modbus data and logs it to a file.

Immediately, within seconds, or runs forever? It’s unclear from that description. :thinking:

The good news is that you can improve it because it’s composed of user contributions. Scroll to the bottom of the documentation page and click the Edit button. You can submit a Pull Request, containing your suggested changes, and a member of the development team will vet it. If accepted, it will be merged into the official documentation.

18

Something that times out is terminated. I used the more active verb because HA actively terminates (or times out if you prefer) the script without telling you it will do that. You only find out in the logs, and even there it is buried in a lot of gobbledygook.

Fair enough, but I thought it was implied, insofar as there would be little point in reading modbus data once. It actually reads it once a minute, and is intended to run forever. Another script aggregates the data every hour into hourly data and also logs that. It is also intended to run forever.

Done, adding a new second paragraph:

“Note that the shell command process will be terminated after 60s, full stop, there are no options to alter this behaviour. This is by design, because Home Assistant in not intended to manage long-running external processes.”

We’ll see what happens…

19

More trouble…this time trying pyscripts. Why am I not surprised it doesn’t work? In fact, I can’t even install it.

I followed the manual installation instructions to the letter:

downloaded latest and unzipped hass-custom-pyscript.zip into config/custom_components/pyscript

added this to config/configuration.yaml:

pyscript:
allow_all_imports: true
hass_is_global: true

check configuration prior to restarting HA only to get:

Configuration invalid!
Integration error: pyscript - Integration ‘pyscript’ not found.

Other custom components I have work.

I’m not going to start a rant because if I do, I’ll probably never stop…

PS I see my suggested addition to the Shell Command documentation has been added, I appreciate the prompt and positive response to my suggestion. Good thing I didn’t start that rant…

20

I think I might have cracked this, based on posts here:

and some key documentation here:

GitHub - aio-libs/async-timeout: asyncio-compatible timeout class.

I am running HA OS on a mini PC, seems to work for this setup, not sure about others because I can’t test on setups I don’t have.

Objective: to run a python script (or scripts) every time HA starts and have the script carry on running in the background (I’m using two to collect process and log modbus data) without being terminated by Home Assassin.

Overview: create a new custom_component x_shell_command and modify it to remove the timeout, which uses async-timeout, which can be disabled by setting the timeout to None. From the above
github link: “timeout parameter could be None for skipping timeout functionality”. Then use x_shell_command with the time out removed instead of shell_command.

No doubt this will be seriously frowned upon, but if it works… I will of course keep an eye on things to see if it goes rogue.

Steps:

(1) create a folder in custom_components called ‘x_shell_command’

(2) download the three files __init__.py manifest.json and services.yaml from here (this is the HA core github) to your newly created ‘x_shell_command’ folder

(3) edit the manifest.json file so it looks like this:

{
  "domain": "x_shell_command",
  "name": "Shell Command (Extended)",
  "version": "1.0",
  "documentation": "https://www.home-assistant.io/integrations/shell_command",
  "codeowners": ["@home-assistant/core","@local"],
  "iot_class": "local_push"
}

(4) edit the __init__.py file lines 17 and 19 as follows:

DOMAIN = "x_shell_command"

COMMAND_TIMEOUT = None

(5) Add the following to the top of your python script. This (apparently) has the same effect as running ‘nohup … &’ on the command line (eg ‘nohup python myscript.py &’) ie no hangups allowed and run in background:

import os
import sys

os.fork() and sys.exit()

(6) Add the following to /config/configuration.yaml (you can add more than one entry as long as the have different names, and note that dot at the start of the path):

x_shell_command:
  run_myscript1: python ./pathto/myscript1.py
  run_myscript2: python ./pathto/myscript2.py

(7) Add the following to /config/automations.yaml:

alias: run_myscript
trigger:
  - platform: homeassistant
    event: start
condition: []
action:
  - service: x_shell_command.run_myscript1
  - service: x_shell_command.run_myscript2

(8) Stop the scripts if they are currently running (via terminal) and then check configuration (Developer Tools > YAML tab, correct if necessary but it should be OK) and if OK restart HA twice, the first time to load custom component x_shell_command, the second to trigger the actions.

I think that is it. The logic is simple enough for even me to follow it. Restart trigger > run custom x shell command action > start python script(s). Any comments welcome.

21

Is another instance of the script executed each time Home Assistant starts? In other words, after restarting twice, are there now two instances of the same script running in the background?

Or is there some mechanism involved that terminates the previous running instance?

22

I’m pretty sure it’s OK, ie there is only ever one instance of the script running at any one time. I’ve checked several times (but it is early days) and never found two instances of either script running. I think this is as expected, because when HA stops it always takes the running scripts with it, because they are running ‘inside HA’, if HA stops, then they stop. I’m going to keep an eye on this, it may depend on whether it is an orderly stop as in a manual restart from Developer Tools or a full blown crash/power cut or other mayhem. But I still incline to think it will be OK, if HA is killed by some malevolent event, it still stops, and (in my reasoning), the scripts stop with it.

23

Here are the running processes after a HA restart, when both scripts were running before the restart:

➜  ~ docker exec -it homeassistant bash
bash-5.1# ps
PID   USER     TIME  COMMAND
    1 root      0:00 /package/admin/s6/command/s6-svscan -d4 -- /run/service
   15 root      0:00 s6-supervise s6-linux-init-shutdownd
   17 root      0:00 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B
   24 root      0:00 s6-supervise s6rc-oneshot-runner
   25 root      0:00 s6-supervise s6rc-fdholder
   33 root      0:00 /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- 
   61 root      0:00 s6-supervise home-assistant
   63 root      0:17 python3 -m homeassistant --config /config
  115 root      0:00 python ./modbus/myscript1.py
  126 root      0:00 python ./modbus/myscript2.py
  127 root      0:00 bash
  133 root      0:00 ps
bash-5.1#

As you can see, only one instance of each script is running after the restart.

BTB, this is the HACS Terminal and SSH, not the ‘Official’ one. The HACS one allows far more access including, as you can see, root access to bash, enabled by the docker exec -it homeassistant bash command.

24

I’ve now devised a way to this that both starts the script(s) when HA starts and restarts the script(s) if they go off-piste while HA is running. The basic idea is to use a command line sensor which detects whether the script is running, and then use a timed interval automation to restart the script if it is not running (which it isn’t when HA restarts, so that also covers the restart).

Note: you can’t use systemmonitor type process to check if the process is running because it uses psutil to get the process name and that only returns ‘python’ not ‘python ./pathto/myscript.py’ and so fails to detect the process.

Follow the steps in my earlier post up to and including step (6) but not step (7). Instead, enable the use of automations.yaml if it is not already enabled by adding this to /config/configuration.yaml:

automation: !include automations.yaml

and then (following the step numbering above):

(7) you need to enable info level logging for the system_log. Add this to /config/configuration.yaml (or just add the logs: entry/line if you already have logger:

logger:
  default: warn
  logs:
    homeassistant.components.system_log.external: info

(8) add the following to /config/configuration.yaml (if you already have sensors, remove the sensor: line below and add the rest under the existing sensor: line):

sensor:
  - platform: command_line
    name: Myscript Running State
    command: "ps -ef | grep myscript.py | grep -v grep | wc -l"

This uses the command line ps tool to check if the script is running, and returns 1 if it is, and 0 if it isn’t.

(9) add the following to /config/automations.yaml

- id: '1680200885274'
  alias: Myscript Running State (On)
  description: Checks if myscript is running
  trigger:
    - platform: time_pattern
      seconds: '30'
  condition:
    condition: numeric_state
    entity_id: sensor.myscript_running_state
    above: 0
  action:
    - service: system_log.write
      data:
        message: myscript.py is running
        level: info

- id: '1680200885275'
  alias: Myscript Running State (Off)
  description: Checks if myscript is NOT running
  trigger:
    - platform: time_pattern
      seconds: '30'
  condition:
    condition: numeric_state
    entity_id: sensor.myscript_running_state
    below: 1
  action:
    - service: system_log.write
      data:
        message: myscript.py is NOT running -> restarting...
        level: info
    - service: x_shell_command.run_myscript

This should be self-explanatory. Each entry needs a unique numeric id, doesn’t matter what it is as long as it is unique. The first automation deals with the is running state, the second with the is not running state. In each case, the time_pattern triggers it to run every 30 seconds after the start of the minute, the condition checks the sensor to see whether it is 0 (not running) or 1 (running) and then takes the appropriate action.

The first automation is just a nicety that gives you logged confirmation the script is running. Bin it if you don’t want it.

(10) do a configuration check and if you get the green light go ahead and restart HA and check all is well. If your code fails the configuration check, note the error(s), correct them, and try again.

25

One way to find a workaround to that limitation of 60 seconds is to spawn the procedure from the shell command… Example for running “test.sh”:

Create a shell command like this (example with arguments passed to the shell script):

shell_command:
   test: /bin/bash /config/test_ini.sh "{{ arguments }}"

The test_ini.sh is very simple (test.sh is the shell script taking more than 60 seconds to run…):

#!/bin/bash
#
#
bash /config/test.sh "$1" &> /config/test.log &

The “_ini” shell will submit (spawn) the test.sh but will not wait the end of that execution of that script and will close immediately after the submission… The test.sh will run like a batch job…
The shell script will execute in the HA container without the 60 seconds limit…

The other way is to submit the procedure in a SSH terminal has explained before (this option even allow to access commands not available within the HA container but allowed/installed in a SSH terminal, see my answer above):

shell_command:
   test: ssh Your_SSH_Username@HA_IP_Adrress -o 'StrictHostKeyChecking=no' -i /config/.ssh/id_rsa '/bin/bash /config/test_ini.sh "{{ arguments }}"'

Make sure your procedure is well tested and will not run for ever while consuming a lot of CPU…:wink:

1 Like
26

@browetd - thanks for adding this, always good to have alternative methods.

I’m pretty sure my method doesn’t cause this sort of problem. There is only ever one instance of the script running (at least so far), and CPU use is as before and stable.

27

I am using the “_ini.sh” “workaround” to read SMS from my modem… The procedure is waiting for a new SMS to come in, so that procedure is running since HA starts and forever… but it is just waiting most of the time…

28

Can you confirm/refute the following thought:

I suspect that spawning the process in the shell created via ssh will mean it will not be terminated when Home Assistant restarts because that shell is executing outside of Home Assistant’s docker container. Therefore each time Home Assistant restarts, it will create another running instance of the script. Only a restart of the SSH & Web Terminal Add-on, or reboot, would eliminate the script(s) running in the background.

Is that right or wrong?

29

@123 To be tested as I never tried to submit via SSH a “never ending” script… I am only doing that in the HA container with only one script (which is (as I mentionned) waiting for a new SMS coming in) and never had any problem in this case…

That’s the reason why I mentionned that you have to be carefull about those workarounds and test them very extensively and seriously before launching it into production…

30

running into a deadend … need some help on this:

I have a .py script that reads some data from an erergymanager and send it via mqtt to HA. on studioCodeServer Terminal the scripts works just as expected.

#
# Copyright: Kilian Knoll, 15.12.2018
# Update: Feb 12 2020: Publish to MQTT
# Utility to parse EnergyManager EM300LR using the JSON API
#
#
#  
#  This program is free software: you can redistribute it and/or modify
#  it under the terms of the GNU General Public License as published by
#  the Free Software Foundation, either version 3 of the License, or
#  (at your option) any later version.
#
#  This program is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#  GNU General Public License for more details.
#
#  You should have received a copy of the GNU General Public License
#  along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
#
#  Please note that any incorrect or careless usage of this module as well as errors in the implementation can damage your Energy Manager!
#  Therefore, the author does not provide any guarantee or warranty concerning to correctness, functionality or performance and does not accept any liability for damage caused by this module, examples or mentioned information.
#  Thus, use it at your own risk!
#
#
#  Purpose: 
#  Returns:
#     Returnvalue: 0    (Everything all right)
#     Returnvalue: -1   (Crap happened)
#     em300data:        (Empty list in case Returnvalue =-1)
#     em300data:        (Full list of key-value pairs in case Returnvalue = 0)
#
# Tested with:
#           python 3.5, 3,7
#           B-control Energy Manager EM300, Firmware Version 2.04
# Based on: https://www.tq-automation.com/Service-Support/Downloads/Downloads-Energiemanagement
# Using the JSON Documentation: https://www.tq-automation.com/content/download/10996/file/TQ%20Energy%20Manager%20-%20JSON-API.0104.pdf
#

# Please change the following values to reflect your environment:
Em_IP_Adress = "192.168.0.23"
EM_Serialumber= "72102414"							#Serial Number can be found accessing the Energy Manager´s Web Page under  Settings - Serial number
EM_Password= ""						    #This is the password you have specified on the Energy Manager´s Web Page (without quotes)
MQTTBroker_IPaddress="192.168.0.118"        #You need to have an MQTT broker set up & running
MQTTBroker_USERNAME="mqtt-user"
MQTTBroker_PASSWORD="2MqTT-User"
# End Configurable Parameters 
#
import requests
import time 
import json
import datetime
import logging      #from loggerdate import loggerdate
from pprint import pprint
import paho.mqtt.client as mqtt
import sys

myparams= {'login': EM_Serialumber, 'password': EM_Password, 'save_login': '1'}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; "
}
mySession = requests.Session()

def changeobiskeys(myjsoninput):
    myjsoninput["Active power_incoming"] =  myjsoninput.pop("1-0:1.4.0*255")
    myjsoninput["Active energy_incoming"] =  myjsoninput.pop("1-0:1.8.0*255")
    myjsoninput["Apparent power-"] =  myjsoninput.pop("1-0:10.4.0*255")
    myjsoninput["Apparent energy-"] =  myjsoninput.pop("1-0:10.8.0*255")
    myjsoninput["Power factor"] =  myjsoninput.pop("1-0:13.4.0*255")
    myjsoninput["Supply frequency"] =  myjsoninput.pop("1-0:14.4.0*255")
    myjsoninput["Active power- "] =  myjsoninput.pop("1-0:2.4.0*255")
    myjsoninput["Active energy-"] =  myjsoninput.pop("1-0:2.8.0*255")
    myjsoninput["Active power_incoming L1"] =  myjsoninput.pop("1-0:21.4.0*255")
    myjsoninput["Active energy_incoming L1"] =  myjsoninput.pop("1-0:21.8.0*255")
    myjsoninput["Active power- L1"] =  myjsoninput.pop("1-0:22.4.0*255")
    myjsoninput["Active energy- L1"] =  myjsoninput.pop("1-0:22.8.0*255")
    myjsoninput["Reactive power_incoming L1"] =  myjsoninput.pop("1-0:23.4.0*255")
    myjsoninput["Reactive energy_incoming L1"] =  myjsoninput.pop("1-0:23.8.0*255")
    myjsoninput["Reactive power- L1"] =  myjsoninput.pop("1-0:24.4.0*255")
    myjsoninput["Reactive energy- L1"] =  myjsoninput.pop("1-0:24.8.0*255")
    myjsoninput["Apparent power_incoming L1"] =  myjsoninput.pop("1-0:29.4.0*255")
    myjsoninput["Apparent energy_incoming L1"] =  myjsoninput.pop("1-0:29.8.0*255")
    myjsoninput["Reactive power_incoming"] =  myjsoninput.pop("1-0:3.4.0*255")
    myjsoninput["Reactive energy_incoming"] =  myjsoninput.pop("1-0:3.8.0*255")
    myjsoninput["Apparent power- L1"] =  myjsoninput.pop("1-0:30.4.0*255")
    myjsoninput["Apparent energy- L1"] =  myjsoninput.pop("1-0:30.8.0*255")
    myjsoninput["Current L1"] =  myjsoninput.pop("1-0:31.4.0*255")
    myjsoninput["Voltage L1"] =  myjsoninput.pop("1-0:32.4.0*255")
    myjsoninput["Power factor L1"] =  myjsoninput.pop("1-0:33.4.0*255")
    myjsoninput["Reactive power-"] =  myjsoninput.pop("1-0:4.4.0*255")
    myjsoninput["Reactive energy-"] =  myjsoninput.pop("1-0:4.8.0*255")
    myjsoninput["Active power_incoming L2"] =  myjsoninput.pop("1-0:41.4.0*255")
    myjsoninput["Active energy_incoming L2"] =  myjsoninput.pop("1-0:41.8.0*255")
    myjsoninput["Active power- L2"] =  myjsoninput.pop("1-0:42.4.0*255")
    myjsoninput["Active energy- L2"] =  myjsoninput.pop("1-0:42.8.0*255")
    myjsoninput["Reactive power_incoming L2"] =  myjsoninput.pop("1-0:43.4.0*255")
    myjsoninput["Reactive energy_incoming L2"] =  myjsoninput.pop("1-0:43.8.0*255")
    myjsoninput["Reactive power- L2"] =  myjsoninput.pop("1-0:44.4.0*255")
    myjsoninput["Reactive energy- L2"] =  myjsoninput.pop("1-0:44.8.0*255")
    myjsoninput["Apparent power_incoming L2"] =  myjsoninput.pop("1-0:49.4.0*255")
    myjsoninput["Apparent energy_incoming L2"] =  myjsoninput.pop("1-0:49.8.0*255")
    myjsoninput["Apparent power- L2"] =  myjsoninput.pop("1-0:50.4.0*255")
    myjsoninput["Apparent energy- L2"] =  myjsoninput.pop("1-0:50.8.0*255")
    myjsoninput["Current L2"] =  myjsoninput.pop("1-0:51.4.0*255")
    myjsoninput["Voltage L2"] =  myjsoninput.pop("1-0:52.4.0*255")
    myjsoninput["Power factor L2"] =  myjsoninput.pop("1-0:53.4.0*255")
    myjsoninput["Active power_incoming L3"] =  myjsoninput.pop("1-0:61.4.0*255")
    myjsoninput["Active energy_incoming L3"] =  myjsoninput.pop("1-0:61.8.0*255")
    myjsoninput["Active power- L3"] =  myjsoninput.pop("1-0:62.4.0*255")
    myjsoninput["Active energy- L3"] =  myjsoninput.pop("1-0:62.8.0*255")
    myjsoninput["Reactive power_incoming L3"] =  myjsoninput.pop("1-0:63.4.0*255")
    myjsoninput["Reactive energy_incoming L3"] =  myjsoninput.pop("1-0:63.8.0*255")
    myjsoninput["Reactive power- L3"] =  myjsoninput.pop("1-0:64.4.0*255")
    myjsoninput["Reactive energy- L3"] =  myjsoninput.pop("1-0:64.8.0*255")
    myjsoninput["Apparent power_incoming L3"] =  myjsoninput.pop("1-0:69.4.0*255")
    myjsoninput["Apparent energy_incoming L3"] =  myjsoninput.pop("1-0:69.8.0*255")
    myjsoninput["Apparent power- L3"] =  myjsoninput.pop("1-0:70.4.0*255")
    myjsoninput["Apparent energy- L3"] =  myjsoninput.pop("1-0:70.8.0*255")
    myjsoninput["Current L3"] =  myjsoninput.pop("1-0:71.4.0*255")
    myjsoninput["Voltage L3"] =  myjsoninput.pop("1-0:72.4.0*255")
    myjsoninput["Power factor L3"] =  myjsoninput.pop("1-0:73.4.0*255")
    myjsoninput["Apparent power_incoming"] =  myjsoninput.pop("1-0:9.4.0*255")
    myjsoninput["Apparent energy_incoming"] =  myjsoninput.pop("1-0:9.8.0*255")
    #serial
    #status
    return myjsoninput

def readenergymanager():
    try:
        #------------------------------------------------------------
        #Start Connection to Energymanager 
            
        #------------------------------------------------------------
        # Initial handshake
        print ("starting initial handshake - start step 1 ...")
        r1 = mySession.get('http://'+ Em_IP_Adress + '/start.php',  headers=headers)
        if (r1.status_code == requests.codes.ok):
            pass
        else:
            Error_Connecting = r1.status_code
            print ("Unable to connect :", Error_Connecting)
    except Exception as Error_ConnecttoEnergymanager:
        print ("Error accessing Energy Manager step1 :", Error_ConnecttoEnergymanager)		
        time.sleep(0.25)
        #------------------------------------------------------------
        # Authenticate with credentials
    try:    
        print ("trying to authenticate -start step 2 ...")
        r2 = mySession.post('http://'+ Em_IP_Adress + '/start.php',myparams,  headers=headers)
        if (r2.status_code == requests.codes.ok):
            pass
        else:
            Error_Connecting = r2.status_code
            print ("Unable to Authenticate :", Error_Connecting)   
    except Exception as Error_ConnecttoEnergymanager:
        print ("Error accessing Energy Manager step2 :", Error_ConnecttoEnergymanager)	        
        #------------------------------------------------------------
        # Get Data from EnergyManager
    try:
        print ("trying to get data - start step 3 ...")
        r3 = mySession.get('http://'+ Em_IP_Adress +'/mum-webservice/data.php',  headers=headers)
        if (r3.status_code == requests.codes.ok):
            pass
        else:
            Error_Connecting = r3.status_code
            print ("Unable to Get data :", Error_Connecting)             
        em300data=json.loads(r3.text)
        Error_Connecting = 0

    except Exception as Error_ConnecttoEnergymanager:
        print ("Error accessing Energy Manager step3 :", Error_ConnecttoEnergymanager)
        #logging.error("%s %s %s", loggerdate(), ",readenergymanager: Ran into exception querying the energymanager : ", Error_ConnecttoEnergymanager)
        Error_Connecting=Error_ConnecttoEnergymanager

        #
        # End Connection to EnergyManager
        #_______________________________________________________________
    # 
    #--------------------------------------------------------------
    # Processing data
    Returnvalue = -1
    if (Error_Connecting == 0):
        if (len(em300data) >1):                         # We have something in the list
            if (em300data["status"] == 0):                  # We got valid data from Energymanager
                ("before calling changeobiskeys")
                em300data =changeobiskeys(em300data)
                Returnvalue =0
    else:                                               # We ran into trouble and allocate an empty list
        em300data={}
        print ("Issue getting data from EnergyManager ", Error_Connecting)
        
    return (Returnvalue, em300data)    
    


if __name__ == "__main__":  
    emclient = mqtt.Client("emclient")                                          #create new instance
    emclient.username_pw_set(MQTTBroker_USERNAME, MQTTBroker_PASSWORD)
    emclient.connect(MQTTBroker_IPaddress)
    
    print ("Start querying Energy Manager....")
    try:
        Myreturnvalue, Mydata = readenergymanager();
        if (Myreturnvalue == 0):
            print ("Returnvalue -should be zero if successful : ", Myreturnvalue)
            print ("----------------Start Values from Energy Manager ----------------")
            pprint (Mydata)
            print ("----------------End - Values from Energy Manager ----------------")	
            print ("Two Specific values from array....")
            print ("Energy to Grid   (Obis code: 1-0:2.8.0)", Mydata['Active energy-'])
            print ("Energy from Grid (Obis code: 1-0:1.8.0)", Mydata['Active energy_incoming'])
            print ("----------------End - Two Specific values from array ------------")   
            params = Mydata.keys()
            print ("----------------Publishing to MQTT  ------------")
            for p in sorted(params):
                print("PublishtoMQTT:","{:{width}}: {}".format(p, Mydata[p], width=len(max(params, key=len))))
                TOPIC = ("Energymanager/"+p)
                emclient.publish(TOPIC,Mydata[p]) 
            print ("----------------END Publishing to MQTT  --------")
        else:
            print ("I was unable to query the Energy Manager")
    except Exception as ex:
        print ("Issues querying EnergyManager :", ex)

But how to get it running on HA-automation calling?

Reading a lot at this community I am on a dead end and need some help from you.

as mentioned at our aproach I created the file: /config/pyscript/run_readenergymanager.py

@service
def run_readenergymanager():
    import sys
    if "/config/pyscript/modules" not in sys.path:
        sys.path.append("/config/pyscript/modules")
    import readenergymanager   
    readenergymanager.mainline()

and /config/pyscript/modules/readenergymanager.py

with:

#
# Copyright: Kilian Knoll, 15.12.2018
# Update: Feb 12 2020: Publish to MQTT
# Utility to parse EnergyManager EM300LR using the JSON API
#
#
#  
#  This program is free software: you can redistribute it and/or modify
#  it under the terms of the GNU General Public License as published by
#  the Free Software Foundation, either version 3 of the License, or
#  (at your option) any later version.
#
#  This program is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#  GNU General Public License for more details.
#
#  You should have received a copy of the GNU General Public License
#  along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
#
#  Please note that any incorrect or careless usage of this module as well as errors in the implementation can damage your Energy Manager!
#  Therefore, the author does not provide any guarantee or warranty concerning to correctness, functionality or performance and does not accept any liability for damage caused by this module, examples or mentioned information.
#  Thus, use it at your own risk!
#
#
#  Purpose: 
#  Returns:
#     Returnvalue: 0    (Everything all right)
#     Returnvalue: -1   (Crap happened)
#     em300data:        (Empty list in case Returnvalue =-1)
#     em300data:        (Full list of key-value pairs in case Returnvalue = 0)
#
# Tested with:
#           python 3.5, 3,7
#           B-control Energy Manager EM300, Firmware Version 2.04
# Based on: https://www.tq-automation.com/Service-Support/Downloads/Downloads-Energiemanagement
# Using the JSON Documentation: https://www.tq-automation.com/content/download/10996/file/TQ%20Energy%20Manager%20-%20JSON-API.0104.pdf
#
# Please change the following values to reflect your environment:
Em_IP_Adress = "192.168.0.23"
EM_Serialumber= "72102414"							#Serial Number can be found accessing the Energy Manager´s Web Page under  Settings - Serial number
EM_Password= ""						    #This is the password you have specified on the Energy Manager´s Web Page (without quotes)
MQTTBroker_IPaddress="192.168.0.118"        #You need to have an MQTT broker set up & running
MQTTBroker_USERNAME="mqtt-user"
MQTTBroker_PASSWORD="2MqTT-User"
# End Configurable Parameters 
#
@pyscript_executor
def mainline():
    import requests
    import time 
    import json
    import datetime
    import logging      #from loggerdate import loggerdate
    from pprint import pprint
    import paho.mqtt.client as mqtt
    import sys

    myparams= {'login': EM_Serialumber, 'password': EM_Password, 'save_login': '1'}

    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; "
    }
    mySession = requests.Session()

def changeobiskeys(myjsoninput):
    myjsoninput["Active power_incoming"] =  myjsoninput.pop("1-0:1.4.0*255")
    myjsoninput["Active energy_incoming"] =  myjsoninput.pop("1-0:1.8.0*255")
    myjsoninput["Apparent power-"] =  myjsoninput.pop("1-0:10.4.0*255")
    myjsoninput["Apparent energy-"] =  myjsoninput.pop("1-0:10.8.0*255")
    myjsoninput["Power factor"] =  myjsoninput.pop("1-0:13.4.0*255")
    myjsoninput["Supply frequency"] =  myjsoninput.pop("1-0:14.4.0*255")
    myjsoninput["Active power- "] =  myjsoninput.pop("1-0:2.4.0*255")
    myjsoninput["Active energy-"] =  myjsoninput.pop("1-0:2.8.0*255")
    myjsoninput["Active power_incoming L1"] =  myjsoninput.pop("1-0:21.4.0*255")
    myjsoninput["Active energy_incoming L1"] =  myjsoninput.pop("1-0:21.8.0*255")
    myjsoninput["Active power- L1"] =  myjsoninput.pop("1-0:22.4.0*255")
    myjsoninput["Active energy- L1"] =  myjsoninput.pop("1-0:22.8.0*255")
    myjsoninput["Reactive power_incoming L1"] =  myjsoninput.pop("1-0:23.4.0*255")
    myjsoninput["Reactive energy_incoming L1"] =  myjsoninput.pop("1-0:23.8.0*255")
    myjsoninput["Reactive power- L1"] =  myjsoninput.pop("1-0:24.4.0*255")
    myjsoninput["Reactive energy- L1"] =  myjsoninput.pop("1-0:24.8.0*255")
    myjsoninput["Apparent power_incoming L1"] =  myjsoninput.pop("1-0:29.4.0*255")
    myjsoninput["Apparent energy_incoming L1"] =  myjsoninput.pop("1-0:29.8.0*255")
    myjsoninput["Reactive power_incoming"] =  myjsoninput.pop("1-0:3.4.0*255")
    myjsoninput["Reactive energy_incoming"] =  myjsoninput.pop("1-0:3.8.0*255")
    myjsoninput["Apparent power- L1"] =  myjsoninput.pop("1-0:30.4.0*255")
    myjsoninput["Apparent energy- L1"] =  myjsoninput.pop("1-0:30.8.0*255")
    myjsoninput["Current L1"] =  myjsoninput.pop("1-0:31.4.0*255")
    myjsoninput["Voltage L1"] =  myjsoninput.pop("1-0:32.4.0*255")
    myjsoninput["Power factor L1"] =  myjsoninput.pop("1-0:33.4.0*255")
    myjsoninput["Reactive power-"] =  myjsoninput.pop("1-0:4.4.0*255")
    myjsoninput["Reactive energy-"] =  myjsoninput.pop("1-0:4.8.0*255")
    myjsoninput["Active power_incoming L2"] =  myjsoninput.pop("1-0:41.4.0*255")
    myjsoninput["Active energy_incoming L2"] =  myjsoninput.pop("1-0:41.8.0*255")
    myjsoninput["Active power- L2"] =  myjsoninput.pop("1-0:42.4.0*255")
    myjsoninput["Active energy- L2"] =  myjsoninput.pop("1-0:42.8.0*255")
    myjsoninput["Reactive power_incoming L2"] =  myjsoninput.pop("1-0:43.4.0*255")
    myjsoninput["Reactive energy_incoming L2"] =  myjsoninput.pop("1-0:43.8.0*255")
    myjsoninput["Reactive power- L2"] =  myjsoninput.pop("1-0:44.4.0*255")
    myjsoninput["Reactive energy- L2"] =  myjsoninput.pop("1-0:44.8.0*255")
    myjsoninput["Apparent power_incoming L2"] =  myjsoninput.pop("1-0:49.4.0*255")
    myjsoninput["Apparent energy_incoming L2"] =  myjsoninput.pop("1-0:49.8.0*255")
    myjsoninput["Apparent power- L2"] =  myjsoninput.pop("1-0:50.4.0*255")
    myjsoninput["Apparent energy- L2"] =  myjsoninput.pop("1-0:50.8.0*255")
    myjsoninput["Current L2"] =  myjsoninput.pop("1-0:51.4.0*255")
    myjsoninput["Voltage L2"] =  myjsoninput.pop("1-0:52.4.0*255")
    myjsoninput["Power factor L2"] =  myjsoninput.pop("1-0:53.4.0*255")
    myjsoninput["Active power_incoming L3"] =  myjsoninput.pop("1-0:61.4.0*255")
    myjsoninput["Active energy_incoming L3"] =  myjsoninput.pop("1-0:61.8.0*255")
    myjsoninput["Active power- L3"] =  myjsoninput.pop("1-0:62.4.0*255")
    myjsoninput["Active energy- L3"] =  myjsoninput.pop("1-0:62.8.0*255")
    myjsoninput["Reactive power_incoming L3"] =  myjsoninput.pop("1-0:63.4.0*255")
    myjsoninput["Reactive energy_incoming L3"] =  myjsoninput.pop("1-0:63.8.0*255")
    myjsoninput["Reactive power- L3"] =  myjsoninput.pop("1-0:64.4.0*255")
    myjsoninput["Reactive energy- L3"] =  myjsoninput.pop("1-0:64.8.0*255")
    myjsoninput["Apparent power_incoming L3"] =  myjsoninput.pop("1-0:69.4.0*255")
    myjsoninput["Apparent energy_incoming L3"] =  myjsoninput.pop("1-0:69.8.0*255")
    myjsoninput["Apparent power- L3"] =  myjsoninput.pop("1-0:70.4.0*255")
    myjsoninput["Apparent energy- L3"] =  myjsoninput.pop("1-0:70.8.0*255")
    myjsoninput["Current L3"] =  myjsoninput.pop("1-0:71.4.0*255")
    myjsoninput["Voltage L3"] =  myjsoninput.pop("1-0:72.4.0*255")
    myjsoninput["Power factor L3"] =  myjsoninput.pop("1-0:73.4.0*255")
    myjsoninput["Apparent power_incoming"] =  myjsoninput.pop("1-0:9.4.0*255")
    myjsoninput["Apparent energy_incoming"] =  myjsoninput.pop("1-0:9.8.0*255")
    #serial
    #status
    return myjsoninput

def readenergymanager():
    try:
        #------------------------------------------------------------
        #Start Connection to Energymanager 
            
        #------------------------------------------------------------
        # Initial handshake
        print ("starting initial handshake - start step 1 ...")
        r1 = mySession.get('http://'+ Em_IP_Adress + '/start.php',  headers=headers)
        if (r1.status_code == requests.codes.ok):
            pass
        else:
            Error_Connecting = r1.status_code
            print ("Unable to connect :", Error_Connecting)
    except Exception as Error_ConnecttoEnergymanager:
        print ("Error accessing Energy Manager step1 :", Error_ConnecttoEnergymanager)		
        time.sleep(0.25)
        #------------------------------------------------------------
        # Authenticate with credentials
    try:    
        print ("trying to authenticate -start step 2 ...")
        r2 = mySession.post('http://'+ Em_IP_Adress + '/start.php',myparams,  headers=headers)
        if (r2.status_code == requests.codes.ok):
            pass
        else:
            Error_Connecting = r2.status_code
            print ("Unable to Authenticate :", Error_Connecting)   
    except Exception as Error_ConnecttoEnergymanager:
        print ("Error accessing Energy Manager step2 :", Error_ConnecttoEnergymanager)	        
        #------------------------------------------------------------
        # Get Data from EnergyManager
    try:
        print ("trying to get data - start step 3 ...")
        r3 = mySession.get('http://'+ Em_IP_Adress +'/mum-webservice/data.php',  headers=headers)
        if (r3.status_code == requests.codes.ok):
            pass
        else:
            Error_Connecting = r3.status_code
            print ("Unable to Get data :", Error_Connecting)             
        em300data=json.loads(r3.text)
        Error_Connecting = 0

    except Exception as Error_ConnecttoEnergymanager:
        print ("Error accessing Energy Manager step3 :", Error_ConnecttoEnergymanager)
        #logging.error("%s %s %s", loggerdate(), ",readenergymanager: Ran into exception querying the energymanager : ", Error_ConnecttoEnergymanager)
        Error_Connecting=Error_ConnecttoEnergymanager

        #
        # End Connection to EnergyManager
        #_______________________________________________________________
    # 
    #--------------------------------------------------------------
    # Processing data
    Returnvalue = -1
    if (Error_Connecting == 0):
        if (len(em300data) >1):                         # We have something in the list
            if (em300data["status"] == 0):                  # We got valid data from Energymanager
                ("before calling changeobiskeys")
                em300data =changeobiskeys(em300data)
                Returnvalue =0
    else:                                               # We ran into trouble and allocate an empty list
        em300data={}
        print ("Issue getting data from EnergyManager ", Error_Connecting)
        
    return (Returnvalue, em300data)    
    


if __name__ == "__main__":  
    emclient = mqtt.Client("emclient")                                          #create new instance
    emclient.username_pw_set(MQTTBroker_USERNAME, MQTTBroker_PASSWORD)
    emclient.connect(MQTTBroker_IPaddress)
    
    print ("Start querying Energy Manager....")
    try:
        Myreturnvalue, Mydata = readenergymanager();
        if (Myreturnvalue == 0):
            print ("Returnvalue -should be zero if successful : ", Myreturnvalue)
            print ("----------------Start Values from Energy Manager ----------------")
            pprint (Mydata)
            print ("----------------End - Values from Energy Manager ----------------")	
            print ("Two Specific values from array....")
            print ("Energy to Grid   (Obis code: 1-0:2.8.0)", Mydata['Active energy-'])
            print ("Energy from Grid (Obis code: 1-0:1.8.0)", Mydata['Active energy_incoming'])
            print ("----------------End - Two Specific values from array ------------")   
            params = Mydata.keys()
            print ("----------------Publishing to MQTT  ------------")
            for p in sorted(params):
                print("PublishtoMQTT:","{:{width}}: {}".format(p, Mydata[p], width=len(max(params, key=len))))
                TOPIC = ("Energymanager/"+p)
                emclient.publish(TOPIC,Mydata[p]) 
            print ("----------------END Publishing to MQTT  --------")
        else:
            print ("I was unable to query the Energy Manager")
    except Exception as ex:
        print ("Issues querying EnergyManager :", ex)

calling the service:

Pyscript Python scripting: run_readenergymanager

i get only the log entry:

2023-12-15 12:49:08.935 DEBUG (MainThread) [custom_components.pyscript.eval] file.run_readenergymanager.run_readenergymanager: calling mainline(, {})

but no new readings

What am I doing wrong? I welcome any form of help to solve the problem and get the script running.

31

@Monster_D Did you found a solution for your problem. Because I’m standing may be on the same script problem. That’s the first time that I’m using phyton scripts on HA.

I use the same python script. It is working on the Sudio Code Server on HA and I get the mqtt messages, thats fine. But if I want to start it by services, I get every time an failure on the protcoll:

ERROR (SyncWorker_14) [homeassistant.components.python_script] Error loading script energymeterreadout.py: Line 202: “name” is an invalid variable name because it starts with “_”

32

Unfortunately I was not able to solve the issue. So for now I am running the script as cron job forum another Linux based system.