Hi,
I am trying to log sensor data to a CSV file from an automation, but I want to create a new file automatically every day.
Right now I am using the File integration with notify.send_message, which works fine for writing to a fixed file. However, it seems like I cannot dynamically change the file path or create new files automatically.
What I want is something like:
A new file every day
Filename format: YYYY_MM_DD_Data_log.csv
Data appended to the file throughout the day
Questions:
Is it possible to dynamically generate file names with the File integration?
If not, what is the recommended way to create and write to new CSV files daily from Home Assistant?
Should I use shell_command, AppDaemon, or something else?
My current setup:
Writing to NAS via /media/log_file/
Using automations for logging data multiple times per day
Any guidance or working examples would be greatly appreciated.
The NAS is mounted as media in the storage.
Thanks!
Probably easiest to use shell_command:
shell_command:
append_to_daily_csv: "printf '{{ text }}\n' >> /media/log_file/data_{{ now().strftime('%Y-%m-%d') }}.csv"
With the above you just need the automation to pass in the text to log.
Why? Doesn’t HomeAssistant store it correctly for you? Are you feeding this into another subsystem, doubling up on storage needs? Should the sensor be writing it to csv directly at data collection time?
Your initial problem seems to be solved by @SJ20035 suggestion, but you may wish to look at the bigger picture, where SQL and MQTT queries, or direct csv writes may be more efficient.
Thank you, I will try it and give an update
It does store it, the problem is that I am using a HA to track some sensors that are creating a lot of data. Around 2.6 million data lines in csv file pr day. In order to be able to handle the files I would like to store them on NAS as daily files. Otherwise the file will increase with 80 mb pr day and therefore be to large to handle.
Maybe there is another way to store it. The data is recieved from several ESP32
You should look at a Time Series Database such as InfluxDB.
That is a lot of data to gather. Not sure if HomeAssistant database may become an issue.
What is your end use?
Why so much data?
Could you store just the data that changes rather than each entry?
That is why I suggested a Time Series Database. You can store data for daily 5 minute intervals and the reduce as the data gets older (ie. Yearly values can be based on Monthly readings - there would be no need for the precision 5 minutes intervals provide).