How to do the most efficient (CPU-friendly) log monitoring -?

[KruseLuds]

I know this was not efficient when I originally set it up, and wanted to revisit it later - and now is the time - but looking for guidance on how to do this… Currently I have several command-line sensors in HA that grep the syslog (and other logs) to monitor activity on my RPI. When the logs get big and several of these firing off simultaneously the CPU briefly spikes, possibly due to 1. each sensor scanning the entire log file each time (some of those doing it at the same time), and 2. the value_template logic I use within the command_line sensors (not as much of an issue I don’t think). Below is an example of one of the sensors, and then my idea/suggestion/ask below on how to do this more efficiently -

Example - I wrote a low-level daemon that turns the fan on if the RPI CPU reaches 60 degrees C and turnes it off when the temperature drops back below 50 degrees C. Every 5 minutes I had that daemon add text like this to the syslog:

Sep  8 16:27:13 kruse-pi fancontrol.py: Fan still off 44.30°<50-60 (1694204833)

The related command-line sensor with this code:

command_line:
  - sensor:
      name: RPI_FAN_STATUS
      command: "grep 'fancontrol.py' /share/syslog | tail -1 | cut -c-255"
      scan_interval: 30
      command_timeout: 5
      value_template: >
        {% set data = value | regex_findall('\(([0-9]+)\)') | first | int | as_datetime | as_local %}
        {% set ts = data.year ~ ' ' ~ (value | regex_findall('(.+)kruse-pi fancontrol') | first).strip() %}
        {% set str = (strptime(ts, '%Y %b %d %H:%M:%S', data) | as_local).strftime('%a %-I:%M:%S%p') %}
        {% set str = str[:-2] ~ str[-2:].lower() %}
        {% set str2 = value[value.find(".py:")+5:] %}
        {% set indx = str2.find(" (") %}
        {{ str ~ ": " ~ str2[:indx] }}
      unique_id: RPI_FAN_STATUS

So I can show this on a dashboard:

Upon researching this, from what I can tell it appears a much better more CPU-friendly way is to run a bash script on the host that monitors the log file using inotify and updates a Home Assistant sensor - directly - when the matching text appears in the log - by using hass-cli?

(BTW the above log entry is the only one where I have control over the process that adds to the log - so I cannot change code in any of the others to send an MQTT message or anything like that to HA…)

So - is the bash script on the host with inotify with hass-cli the best way to most efficiently get log data into an HA sensor (I am always looking for the most recent instance of a string when it appears in the log) - and how the heck to I do that - ? or is there a way to have a sensor which would just continuously tail the log directly from within HA - would that be even better (and easier) and how would I do that (I thought I could do something through the recorder but that’s not going to work I don’t think)?

Any ideas would be very helpful - ?

(P.S. I just checked some of my logs, they are bigger now than I have ever seen them so I cleaned them out due to other unrelated recent errors that were in them - and the problem is not as much of an issue now - but I still do want to get rid of all of these GREP statements - )

[lepton]

The most efficient “sensor” is going to be building an integration that just reads that file into a pipe and sets the fan status based on that.

[KruseLuds]

Thank you @lepton - that makes sense - and the pipe should have the ability to look for multiple things at the same time, so it could update several sensors… I was going to ask how I do that but I tried ChatGpt and here is the output. Please let me know if you think that would work better than the icnotify/hass-cli approach - and if you see anything wrong with what CHatGPT has suggested - ?

What my request was to ChatGPT:

Write the home assistant code for an integration that continuously reads the output of a syslog into a pipe to be used for notifications

ChatGPT

Creating a Home Assistant integration that continuously reads the output of a syslog and uses it for notifications requires a combination of Home Assistant configuration, a Python script for reading syslog, and possibly some custom components or automations. Here’s a high-level guide on how you can achieve this:

1. Python Script for Reading Syslog:You can use Python to read the syslog and send events to Home Assistant. Here’s a basic script that reads syslog and sends messages as events to Home Assistant using the hassapi module:

python:

import os
import sys
import subprocess
import time
import hassapi as hass

class SyslogReader(hass.Hass):
    def initialize(self):
        syslog_file = '/var/log/syslog'  # Adjust this path to your syslog file
        self.process = subprocess.Popen(['tail', '-F', syslog_file], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

        while True:
            line = self.process.stdout.readline()
            if line:
                self.log(line.strip())
                self.fire_event("syslog_message", message=line.strip())
            else:
                time.sleep(0.1)

Save this script as syslog_reader.py in your Home Assistant configuration directory.

2. Set Up Home Assistant Integration:Add the following lines to your Home Assistant configuration (usually in configuration.yaml or split into separate YAML files):

automation:
  - alias: Notify on syslog message
    trigger:
      platform: event
      event_type: syslog_message
    action:
      service: notify.notify  # Replace with your notification service
      data_template:
        message: "Syslog Message: {{ trigger.event.data.message }}"

Replace notify.notify with the actual notification service you want to use (e.g., notify.notify, notify.mobile_app, etc.).

3. Run the Python Script:Run the Python script using AppDaemon, which is a supported Home Assistant add-on. Make sure you have AppDaemon installed and configured in Home Assistant.

4. Restart Home Assistant:After configuring the automation and running the Python script, restart Home Assistant to apply the changes.

5. Test the Integration:Generate a syslog message on your system (e.g., by running a command that generates syslog messages), and Home Assistant should trigger the automation and send the notification.

Please note that this is a basic example, and you may need to adapt it to your specific use case and syslog setup. You can also add additional logic to filter or process syslog messages as needed. Additionally, ensure that the hassapi module is properly installed and configured in your AppDaemon environment.

Remember to adjust the syslog file path and customize the automation and notification services according to your setup and preferences.

@lepton Is the above the most CPU-efficient approach?

[lepton]

The most efficient method would be to add an instruction to your CPU for this task, but this is a reaonable approach.

[KruseLuds]

Can you elaborate - you mean a daemon running the python code on the host instead of being fired up by HA, and then utilizing hass-cli or mqtt to get it into HA, or ?

[lepton]

I was making a joke, because it doesn’t matter which approach you choose as long as it matches the ChatGPT scheme.

[KruseLuds]

Sorry, Mr Gullible here, I’ll give it a shot.

Better to have the script launched by HA, and only a minute or so after it is all up and running (when the SYSLOG calms down) as it would only be needed by HA, and not using notify but just direct calls to populate all the other sensors as appropriate with the correct values with functions as shown below -

I am fairly new to this, but I think it’s safe to say that this would run faster than a longish if statements:

def case1_function():
    # Do something for case1

def case2_function():
    # Do something for case2

def case3_function():
    # Do something for case3

# Create a dictionary mapping options to functions
cases = {
    'case1': case1_function,
    'case2': case2_function,
    'case3': case3_function,
}

# Call the appropriate function based on the option
option = 'case2'
if option in cases:
    cases[option]()
else:
    # Default case

…or would that not be useful for speed purposes as there are ~20 sensors I am updating - ?

[lepton]

Computers don’t care about 20 of anything. Call me when you have 2 million.

[KruseLuds]

@lepton can you help with the below, which is my only hurdle in implementing this (updating the input_text - sample as shown in the text)?

**ARGH**&#@^%$##$% I must be blind, why this error on such a simple request?

I have a simple file named test.py: from requests import post url = "http://localhost:8123/api/states/input_text.my_data" headers = {"Authorization": "Bearer < then my huge long token is here >", "content-type": "application/json"} data = {"state": "25"} response = post(url, headers=headers, json=data) print(response.text) …and when I said above: < then my huge long token is here > I just meant that is replaced with just the hugely long string I have generated in home assistant as a long…