Data from retained messages causing data spikes in MQTT sensors

Dear clever people,

I’m hoping someone can point me in the right direction. I have an OVMS (open vehicle monitoring system) module in my car which sends metrics via MQTT to my Home Assistant instance via the Mosquito add-on. I have it going via HiveMQ as that was the easiest way I could get it to integrate securely and use it whilst away from my home WiFi, using LTE. It’s all working well but I have ongoing issues with gaps in data, presumably caused by transient availability issues, which then result in transient data spikes to what is presumably a retained MQTT message as the y-axis values of the spikes are usually identical. The OVMS module doesn’t give me any configuration options, but seems to send QOS 0 messages (looking at MQTT explorer and HiveMQ). I have the sensors set up using yaml in HA.

These spikes cause problems further down the line with automations/calculations etc. If I publish a blank message to the relevant topic to “clear” the retained message it sorts things out for a while but then recurs. I’ve scoured the documentation and forums and experimented with all sorts of templates and availability settings. The only thing I’ve found that works reliably is to set up a second trigger template sensor as below. It works but isn’t very elegant and seems like using a sledge hammer to crack a nut. Is there a simpler approach I could use within my MQTT yaml to filter these states out please?

here is an example of a working MQTT sensor

  - name: MG ZS SoC MQTT
    unique_id: mg_zs_soc_mqtt
    state_topic: ovms/xxxx/xxxx/metric/v/b/soc
    unit_of_measurement: "%"
    device_class: battery

And here is the template that works:

- trigger:
    - platform: state
      entity_id: sensor.mg_zs_soc_mqtt
      not_to:
        - unknown
        - unavailable   
        - none 
      not_from:
        - unknown
        - unavailable   
        - none                      
  sensor:
    - name: "MG ZS SoC"
      unique_id: mg_zs_soc
      unit_of_measurement: "%" 
      device_class: battery
      state: >-
          {{ states('sensor.mg_zs_soc_mqtt') | float(1) | round(0) }}

And here’s an example. All the raw MQTT sensors show a similar spike at identical times when it happens.

Any suggestions for how I could improve my MQTT yaml would be gratefully accepted!
Thanks

Try enabling LWT on your mqtt broker. This should mark the topics as unavailable if the client is disconnected.

Thanks for your reply. I can’t work out how to do this on my broker (Mosquito addon running on HA OS) but looking in the HA MQTT integration settings will messages are enabled already. Is this what you’re referring to? Sorry if I’m being thick.

I thought you were using HiveMQ?

Sorry Tom, I should have clarified earlier. I have an MQTT bridge between Mosquito (as HAOS add-on) as my main broker, and the free tier of HiveMQ Cloud as a remote broker. I originally set it up like this as it seemed the easiest way of getting a (relatively) secure connection without the worry/hassle of opening up remote access to my locally hosted mosquito broker. It was easy to set up and has otherwise worked very well. I have set it up (I think) to mirror everything across from one to the other. Here is the .conf file for Mosquito:

# bridge from hivemq to mosquito, for OVMS MQTT server
log_type all
connection hivemq
address xxx.s1.eu.hivemq.cloud:8883
remote_clientid hassio
remote_username xxx
remote_password xxx
start_type automatic
bridge_cafile /share/mosquitto/isrgrootx1.pem
cleansession true
notifications false
try_private false
bridge_insecure false
topic # both 0

In case anyone is interested, I resolved the issue of (often old) retained messages causing spikes following periods of unavailability. I had the bridge between HiveMQ Cloud broker and Mosquito broker incorrectly configured. Once I corrected these settings it has all worked much better:

try_private true 
bridge_protocol_version mqttv50