Logger into HA file by default?

ESPHome
1

I have … a kinda strange idea:
I have numerous ESP modules, many of them are built by myself, so occasional errors or not-correct functioning is not exactly a stranger to me, it can be due to bad programming (i’m not a pro here…), but mostly because either bad hardware design, missing component (pull-up etc…), or anything similar.

Now, i’d like to debug them as much as i can, so, what i wonder is - is it a chance to save logger from esphome modules into log files by default? I know that i can trigger writing single module’s log into file my myself via ssh (with command: “esphome logs kitchen-dimmer.yaml > kitchen_dimmer.log”), but i’d like for all my modules to write such a log by default - each module into a their own file.
It will be quite some MB of data, i know, but i have HA in VM on NAS, so i can assign HDD size as much as i wish, and i’d try to make an automation which would delete those files, say, each week or so.

Does anyone knows if this would be possible and how?

2

The only way I can think of doing it is using on_message: and using say a http post or some other method to write the data. You obviously would need some process/server to receive the data but it wouldn’t need to be part of HA.

3

Aha… you mean to create a text sensor in ESPHome which would send messages to HA? Well, i can try to play with it if something usefull comes out… thanks for idea!

4

No. on_message: is part of the logger component and is triggered for every log message. The example shows it sending a mqtt message but you could just as well send a http message to a web server.

5

Here is a link to an open source mqtt logger written in Python.

6

Thanks a lot! It’s quite some things to study now… i’m not exactly “friend” with mqtt, so i’ll have to do some digging here. An idea to send http message to a (file on the) web server seems great, but i must figure out how to do it, so i’m back into google now…
do you have any quick suggestions/examples?

7

A quick recap after first testing: http is a no-go, since: first: it doesn’t support esp-idf, second: fw file for esp is too big, so it won’t flash over OTA - i’d need to have an esp module with bigger memory (i’m testing on ESP32).
I also succesfully created text sensor, but i wouldn’t to in this direction either, since i’d pump up HA’s database file into heaven, i guess…

So, to mqtt testing…

8

I could probably whip something up in Node-Red fairly quickly.

It has the ability to accept a HTTP POST or an MQTT topic as an input, process it and the write it to a file. Or several files based on the contents of the input.

Have a fiddle (wine o’clock for me) and see if you can nut something out, otherwise may have time to assist tomorrow evening.

Note that a solution with Node-Red requires your HA server to be up and running - whereas an external server would work regardless.

EDIT: It’s getting past evening here now for reference :smiley:

9

Oh - and an example of sending an http message from ESPHome. This is to another ESPHonme device - but you get the drift.

binary_sensor:
  - platform: gpio
    pin: GPIO12
    id: toggle_pin
    name: "Toggle"
    on_press:
      - http_request.post:
          url: http://weather.local/light/weather_backlight/toggle

This is just a call to the ESPHome API, but you could send any message - read the docs:

10

Many thanks for your help!
As i said: http won’t go, as it seems, since fw file won’t fit into even my 4MB esp32, not to mention that i have quite a few esp8266 modules with only 1MB…
I’l dig into mqtt and your mqtt logger suggestion.
(BTW…it’s before noon here :slight_smile: )

11

Wow! What else is in your code??? Web server does chew a bit up but I didn’t think http_request: was that hungry…

I reckon the Node-Red solution would be the best - it can write to an accessible path on HA without clogging up your DB.

12

I have CO sensor (MQ7) and a couple of relays on the same board (it’s still a testing board at this point, though), perhaps co component is also big…
If you’ll have some time then i’d appreciate if you write node-red example for me, it’s no hurry, though…
I don’t use node-red, but i’l install it (and learn a bit) if necesarry, no problem. But, as said, take your time - i have plans for these loggings for quite a while anyway…

13

How about something like this? You do need a syslog server somewhere:

2 Likes
14

Wow… this looks excellent! I DO have syslog server - i have synology NAS. And it’s already logging my test esp! So simple, yet so “complicated” when a man doesn’t know how to…

First test are just what i wanted - i’ll dig into this a bit more for details.

Many thanks!
@zogara: many thanks to you, too for your help and support! Wait a bit with node-red for a while to see if this really works as planned.

EDIT: ok, it seems that this addon works with arduino only, too. I have some esp-idf modules (merely beacuse of bt-proxy, which supposely works better with esp-idf), but i guess i can temporarily switch them to arduino if i’ll have problems with either one of them.

1 Like
15

Hopefully there’s an idf version out there, as syslog is a defacto standard. My main reason for mentioning was to point you to syslog as an option over file.

17

Well, as i found on original library’s github page it requires arduino. But as you said: perhaps esp-idf version will (eventually) be available (or discovered).
For now my main reason for needing this is solved: i have occasional non-explainable light turn-on in my sleeping room and i have to find out why this is happening. HA’s log doesn’t tell much, only that one binary sensor was turned on (which i already know…). Perhaps this log will help, although i have a suspicion that pull-up resistor in my wall button is either missing or it’s too big value. Since there’s a few metres from module to button there’s a possibility for a wire to catch all kind of “dirt” spikes and trigger input… If i’ll have a log confirmation which will say that button was indeed pressed i’ll know for sure that this is the cause.

18

I created a tutorial with a shell script to automatically start and stop esphome logs for all your devices that creates a separate full colour log file for each device - TUTORIAL: ESPHome File Logger - Works with HAOS

If you really want the logs to appear in the HA log file you can use the safe_log function as a starting point and take each line as it comes, and use a regex to modify it to the HA log format before appending to the HA log file. You can simply the script as you don’t need the ANSI colour codes either.