I’m pretty new at Home Assistant, so maybe I have just not found the correct module, but it seems to me that there is no way to write out history to a text (log) file? Is that so, or is there something I’m missing?
The reason I would like a log file is to be able to move data on to other targets, primarily Elasticsearch via Filebeat. I have found a workaround using the graphite History plugin in Home Asisstant, and the Graphite input for logstash, but this is quite a hack and does not make use of the local filesystem and back-off capabilities of Filebeat…
Is there any way to output the stream of History events to a plain text file (one event per line, preferably each line JSON formatted)?
You might try looking at MQTT EventStream or MQTT Statestream components to capture the data. ElasticSearch seems to have some ways to receive MQTT messages.
That’s interesting and I’d thought of that too, but there’s no easy way to get MQTT into logstash… just pushing the problem one step downstream…
Maybe this can help you out? Logger - Home Assistant
As far as I can see, this logs about the modules, but not the actual event data, and also it logs everything to a single file - there is no way to separate debug logging on the zwave module from the actual event data into separate files…
I am sorry, I thought your target was ElasticSearch and this would provide the bridge. But I guess I misunderstood.
i believe to have found a way:
If i understand correctly you want the logbook functionailty built in to homeassistant but then as a plain text log file?
If you check the network tab in chrome developers tools (f12) on the logbook page. you can see the request that the client does to the Home assistant API.
this gives all state changes in JSON format as a result:
You can do this API call yourself with the restfull API included in home assistant:
I am sorry, I thought your target was ElasticSearch and this would provide the bridge. But I guess I misunderstood.
True - my end-target is Elasticsearch, but I’m not sure I’ll want to put in all the data without manipulation first. Great idea though - I’m grappling with getting mqttbeat compiled now to try it out. 
Just wanted to share my experiences of getting this set up. Thanks gpbenton for the suggesting Statestream, it was the piece I had been missing.
Getting mqttbeat compiled wasn’t too challenging. If you aren’t used to pulling and compling golang apps then you might be tempted to pull with git. If you do this then make will complain about missing dependencies. There are various guides around for getting the go environment set up. In short I did:
go get github.com/elastic/beats
go get github.com/nathan-k-/mqttbeat
cd src/github.com/nathan-k-/mqttbeat/
make
I am using Logstash as the output so set up the endpoint settings per the example mqttbeat.yml file. Nothing too special here but I chose to set a tag of “mqttbeat” to identify the messages. More on that later.
tags: [“mqtt”]
Now I configured statestream in homeassistant. If you omit the include then homeassistant will output a lot of data. I am only interested in the sensor data to create visualisations and limiting the output makes it a lot simpler to process the resulting data.
mqtt_statestream:
base_topic: homeassistant
publish_attributes: false
publish_timestamps: false
include:
domains:
- sensor
If you configured everything correctly you should start to see data appearing in Kibana. This is not quite the end of the story as Kibana will identify the payload from mqttbeat as a string. I might have messed up here because I didnt import any template. I use a logstash mutate to convert the payload to a float.
One additional complication when examining the JSON from mqttbeat is that it doesn’t set the type field to the beat name (I havent tried adding the field to the beat config yet which might be an option). Remembering the tag we set earlier, you can create the following filter using the tag.
filter {
if "mqtt" in [tags] {
mutate {
convert => { "payload" => "float" }
add_tag => [ "sensor" ]
}
}
}
Now going back to Kibana you should be able to set the payload field index pattern to float (if the index exists then you may need to delete it first). Now go create those pretty graphs!
I hope you will find these pointers useful to get yours working. They would certainly have saved me some time 
2 Likes
Thank you for the super write-up Russel! 