What I want to achieve now is to have the mqtt session to be be persistent (described here Persistent Session and Queuing Messages - MQTT Essentials: Part 7) so in case that home assistant is down due a reboot the data will still be received when is back up again.
Is this possible? I tried searching through the documentation of home assistant but could not find anything.
You need what ever is sending the message to mark it as retained so it is kept on the broker. Then when HA comes back online there is a message stored for it to read
to my understanding the retain flag is only good so the subscribing client (in this case the home assistant sensor) receives immediate the “last known good value” that the broken has saved.
In my case however I want the subscribing client to received all the messages that broker did not sent because the subscribing client (home assistant) was down. For example lets say that my subscribed client gets an update every time my power changes its wattage value. This wattage is in turn being accumulated in another sensor in order to have total kWh consumption. If I loose updates on the sensor that measures power in watt then the total accumulation of the sensor is wrong.
I know that mosquito broker can have subscribing clients that are persistent and I already configured my publishing client so they sent their values in persistent mode. What I am asking is if it is possible for home assistant mqtt integration to handle persistent sessions to the topic so I can not loose values when home assistant is down for some reason.
No the addon supports this. Here’s a mosquitto specific guide on this. As you can see the only requirement for the broker to support persistent sessions is to have persistence true in the config. Everyone running the mosquitto addon has this, it’s right here in the config:
To use a persistent session clean_session must be false. But HA does not specify it therefore it is always going to default to True.
@phanosp you’ll have to make a feature request for this. That being said, you’ll also have to explain what you would expect this to do. The sensor’s state should always be the latest value so what are you expecting HA to do with the stale messages? They wouldn’t be used to set the state of the sensor as that would be incorrect by the time they’re received.
Ah just answered my own question, found the use case. So I guess what you would be looking for the queue of messages to be processed by changing the sensor’s state multiple times in rapid succession. This way your utility meter (or whatever your accumulator sensor is) would see all those state changes and accumulate all of them.
Well definitely make sure to mention this use case in the feature request. I’m not sure how feasible it is but perhaps someone will figure it out.
I understand why phanosp wants the MQTT Sensor’s topic to receive a queue of all values that occurred while Home Assistant was offline (or merely disconnected from the MQTT broker). However, persistence would apply for all subscribed topics and here’s how it would affect, for example, an MQTT Binary Sensor for a door or motion sensor.
While Home Assistant is disconnected from the broker, the broker creates a queue containing every on/off state (open/close) of the motion sensor.
The moment Home Assistant re-connects to the broker, it would receive the queued payloads and rapidly toggle the binary_sensor’s state.
An automation monitoring the binary_sensor would be triggered repeatedly and perform actions in succession that were never anticipated when it was designed (like sending a flurry of consecutive notifications or toggling devices repeatedly).
Hi Mike and thanks for clarifying things up. I was suspecting that the mqtt integration was not supporting persistent session but I was not sure. Basically what I want is what you explain on your second post. HA should receive all messages for a certain period which it was down/offline and a utilitiy meter should accumulated them. Not sure if the utility meter will be able to do that since to my understanding in order to accumulate “correctly” all the info the time frames should also be consistent. I mean if my wattage was 500 watt for 5 minutes and then its was 1500 watt for 1 minute and then 500 watt again the utility meter should take this into consideration.
To my understanding the broker should include timestamps so utility meter in HA should not have a problem with this but I am not sure. If someone can clarify things this will be better.
Regarding Taras comment I was under the impression that I can have persistent mode in MQTT per topic not necessarily for everything and everyone that is subscribe to the mosquito broker. For example have something on the sensor like
This way the problem you mention in your post should never occurred unless you specifically ask for that topic to be persistent.
Not sure 100% if this how it will work. Maybe someone can also clarify things here as well. If what I mention above is correct I will continue to make a feature request
A timestamp for the MQTT message would allow you to reconstruct the timeline to perform the integral calculation needed. But it would also require the MQTT integration to pass that timestamp on to the recorder as the timestamp of the change, and I very much doubt that. But that would be nice indeed, if not for all the side effects of a very long down time. Automations would need to be suppressed.
I was under the impression that persistent topics would replace the previous ones for that topic. So what you are saying is that is not the case per se for persistent clients, as they will queue them until read?
I also do not think the utility_meter sensor is capable of converting from W to kWh. Are we talking about a different sensor, or is this just a thought experiment? I’m constantly amazed that there seems to be a sensor type for everything, but I cannot always find the right one that easy.
the utility meter does not convert W to kWh directly. First I get the watt in a sensor of platform: mqtt (current consumption), then I use a sensor of platform: intergration to accumulate the wattage. Then the platform:integration is passed to a utility meter to get the value.
I do not pass the timestamps anywhere so I am not sure if they are actually needed to be pass directly or somehow are being done by the mqtt integration.
Another setup that I have I am sending the values to an influxdb and then I can graph the data using grafana. In this case I do not pass timestamps and influxdb takes cares of the timestamps. Not sure though 100% maybe I am missing something here. Maybe influxdb (being a time series platform) simply adds the timestamps when the message is received.
True told although that I am using persistent in the influxdb/grafana system I am not sure if any data received after the system was down have any advert effect and those data are being record incorrectly. Since both grafana/influxdb and mosquito broker are on the same machine I can not really test it through to be sure.
Maybe someone with more knowledge can clarify things more about how does the mqtt integration in home assistant and platform:integration work here.
All state changes are timestamped. The sensor uses the timestamps that are recorded to reconstruct the timeline. So that is why the sensor can do the calculation.
But the main question is, does the recorder use the timestamp now() when it receives a state change, or can the integration (MQTT) specify one of its own when applying the state change. And if it can, does it? I suspect the recorder will make up the timestamp on its own. In that case MQTT processing the queue in rapid succession will not fix the downtime period properly.
I see what you mean. If all messages that will be received though the broker due to downtime will have the same timestamp that will create many more issues and solve none I think.
In a perfect world messages sent to the mosquito broker should (at least in theory) include the timestamp of the client who published it at the time it sent it. If this is in the specification of mosquito broker then the mqtt integration of Home Assistant should be able to implement this as well.
But I do not have more knowledge on the matter. I will try to research little bit more before I make a feature request. Maybe in the meantime someone else with more knowledge on the matter can shed more light on this.
The documentation link you provided in your first post explains it is a persistent session. The term “session” in this context means from the moment a client connects to the broker to when it disconnects. All of the client’s subscriptions during the persistent session are stored.
Which part of the documentation led you to believe you could specify persistence by topic?
my comments come from experiencing with the command line mosquito_sub. When a client (a single command/client subscripts to a topic), the user can provide the ability for that command/subscription to be persistent and with a unique client id.
So as I understand it you can have more than one subscribing client of the same machine with different topics and client_ids. For example
I already clarified it in my previous post but if it failed to help you understand the concept, I suggest you review HiveMQ’s documentation or this one:
It’s not true. Or rather @phanosp didn’t mention session in his the first post different way than mentioning MQTT essentials docs.
For sake of clarifying the feature: the “persistent session” feature is to collect generated messages even if the client has disconnected, in order to deliver those messages later.
The persistent session is initialized by connecting with cleanSession = false. And it will collect messages in case the client has disconnected. To disable queuing messages, the client has to connect with cleanSession = true. It’s even mentioned in docs you linked.
But also it’s true that the persistence is not set by topic. But by client id.
However… it doesn’t mean HA cannot make 2 connections to MQTT, one with and one without persistence
Phanos first post does use the term “mqtt session” in the same manner as HiveMQ’s documentation for “persistent session”. However, in a later post, they make the assumption that persistence can be per topic.
TBH I had no problem understanding his intention. I’m not even sure using ‘mqtt session’ in his case is totally wrong. Especially in the context of the persistence feature.
It’s not true either. It’s not per session, It’s per device ID. By connecting and disconnecting you can create multiple different sessions (depending on level of networking we taking into consideration). However if all those sessions manifest the same device id, it allows creating persistent session within mqtt protocol.
When a client connects to the broker, the session it establishes can be either “clean” or “persistent”. For example, Home Assistant establishes clean sessions.
What is this “deviceID” you are referring to? If you mean “clientID” (like in HiveMQ’s documentation) then that serves to identify the one session from another (a client can establish multiple concurrent sessions with the broker).
By connecting and disconnecting you can create multiple different sessions (depending on level of networking we taking into consideration). However if all those sessions manifest the same device id, it allows creating persistent session within mqtt protocol.