MQTT Problem Popped Up Recently

sparkytech · July 23, 2023, 2:35am

I’m not sure what changed, but I need some guidance after much trial and error. I am running CumulusMX weather station software which outputs data to MQTT. This was working for a long time, and I checked it today, and its no longer harvesting data into HA. MQTT Explorer still shows data coming into the MQTT server in HA, but I think maybe I have a syntax problem with my sensors.

CumulusMX MQTT Config (for reference)

{"topics": [
    {
        "topic": "CumulusMX/DataUpdate",
        "data": "\"time\":\"<#timehhmmss>\",
		\"windspeed\":\"<#wlatest rc=y>\",
		\"winddir\":\"<#bearing>\",
		\"wgust\":\"<#wgust>\",
		\"rfall\":\"<#rfall>\",
		\"rrate\":\"<#rrate>\",
		\"LastRainTipISO\":\"<#LastRainTipISO>\",
        "retain": false
    }
]}

Typical Home Assistant MQTT Sensor:

  - name: "Davis Windspeed"  
    state_topic: "CumulusMX/DataUpdate"
    unit_of_measurement: 'mph'
    expire_after: 30
    icon: mdi:tailwind
    value_template: '{{ value_json.windspeed }}'

Any ideas? Thanks in advance!

dproffer · July 23, 2023, 4:38am

Hard to tell due to the way you presenting the information in this post, however ‘DataUpdate’ topic does not look to be a valid JSON string. Also, any time I see a bunch of ‘quoted’ double quotes around JSON, flags go up.

First stop might be to see if MQTT Explorer thinks your ‘DataUpdate’ topic is value JSON.

JSON can be a PIA, take it one step at a time in your debugging. Good hunting!

koying · July 23, 2023, 7:33am

Actually, the first stop is always to check the HA log for errors

It likely shows something like “value_json” is null, because, indeed, “DataUpdate” is (no more?) a valid JSON object.

sparkytech · July 23, 2023, 2:12pm

Thanks for the replies! It was late and I forgot to mention, after MQTT quit working, the Docker container that was running this CumulusMX weather software was migrated to an LXC container for simplicity and troubleshooting. In the process, the CumulusMX software was updated, and the MQTT files (Data and Interval files) syntax also changed slightly with the update. The way I got this halfway working yesterday was to play around with the quotes in the string until it showed up again in MQTT Explorer. The way it’s currently formatted was what I settled on, as it was at least halfway working. I was also thinking something was still slightly off with the string, I’m just not sure what is incorrect.

I’m not sure what you mean by:

“…Also, any time I see a bunch of ‘quoted’ double quotes around JSON, flags go up.”

-and-

“It likely shows something like “value_json” is null, because, indeed, “DataUpdate” is (no more?) a valid JSON object.”

Here is the original “Interval Template” file:

{"topics":[
    {
        "topic": "CumulusMX/Interval",
        "data": "{\"time\":\"<#timehhmmss>\",\"temp\":<#temp rc=y>,\"humidity\":<#hum>,\"wgust\":<#wgust rc=y>}",
        "retain": false
    }
]}

And the original “Data Update” file:

{"topics": [
    {
        "topic": "CumulusMX/DataUpdate",
        "data": "\"time\":\"<#timehhmmss>\",\"windspeed\":<#wlatest rc=y>,\"winddir\":<#bearing>",
        "retain": false
    }
]}

Here are the corresponding log entries:

2023-07-22 22:29:55.985 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.temp }}'
2023-07-22 22:29:55.999 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.apptemp }}'
2023-07-22 22:29:56.003 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.intemp }}'
2023-07-22 22:29:56.008 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.cloudbasevalue }}'
2023-07-22 22:29:56.011 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.humidity }}'
2023-07-22 22:29:56.013 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.inhum }}'
2023-07-22 22:29:56.017 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.pressure }}'
2023-07-22 22:29:56.021 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.dewpoint }}'
2023-07-22 22:29:56.024 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.wchill }}'
2023-07-22 22:29:56.028 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.heatindex }}'
2023-07-22 22:29:56.035 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.humidex }}'
2023-07-22 22:29:56.039 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.avgbearing }}'
2023-07-22 22:29:56.043 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.domwinddir }}'
2023-07-22 22:29:56.046 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.Tbeaufort }}'
2023-07-22 22:29:56.050 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.windrun }}'
2023-07-22 22:29:56.054 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.cloudbasevalue }}'
2023-07-22 22:29:56.059 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.SensorContactLost }}'
2023-07-22 22:29:56.065 ERROR (MainThread) [homeassistant.components.mqtt.models] Exception raised when updating state of sensor.cumulusmx_interval, topic: 'CumulusMX/Interval' with payload: b'{"time":"20:58:43",\r\n\t\t"battery":ok,\r\n\t\t"press":29.26,\r\n\t\t"pressure":Rising,\r\n\t\t"inside-temp":69.3,\r\n\t\t"temp":74.1,\r\n\t\t"tempTL":74.1,\r\n\t\t"tempTH":83.3,\r\n\t\t"humidity":62,"inside-humidity":55,\r\n\t\t"wgust":4.0,\r\n\t\t"dew":60.2,\r\n\t\t"dewpointTL":58.4,\r\n\t\t"dewpointTH":63.5,\t\r\n\t\t"apptemp":76.9,\r\n\t\t"apptempTL":0.0,\r\n\t\t"apptempTH":87.3,\r\n\t\t"wchill":74.1,\r\n\t\t"wchillTL":0.0,\r\n\t\t"wchillTH":,\r\n\t\t"heatindex":74.1,\r\n\t\t"heatindexTL":,\r\n\t\t"heatindexTH":84.4,\r\n\t\t"humidex":27.7,\r\n\t\t"wlatest":'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/models.py", line 270, in process_write_state_requests
    entity.async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 742, in async_write_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 867, in _async_write_ha_state
    hass.states.async_set(entity_id, state, attr, self.force_update, self._context)
  File "/usr/src/homeassistant/homeassistant/core.py", line 1651, in async_set
    state = State(
            ^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 1269, in __init__
    raise InvalidStateError(
homeassistant.exceptions.InvalidStateError: Invalid state encountered for entity ID: sensor.cumulusmx_interval. State max length is 255 characters.

koying · July 23, 2023, 4:50pm

The error tells you’re now trying to put the full value into a state.
We cannot help you if we don’t have the latest version of your code.

The initial issue is still that DataUpdate is not a JSON string. JSON starts and end with curly braces ("{" and “}”), as in the “Interval” topic.

JSON is recognized as such in MQTT Explorer, so you cannot be mistaken

sparkytech · July 23, 2023, 5:41pm

Alright, I think I’m making progress, thanks for the correction. Here are my current MQTT files, which I shortened for simplicity. I checked both of these in JSONLint for accuracy, and they now pass:

DataUpdate Template:

{
	"topics": [
		{
			"topic": "CumulusMX/DataUpdate",
			"data": "{\"time\":\"<#timehhmmss>\",\"windspeed\":<#wlatest rc=y>,\"winddir\":<#bearing>,}",
			"retain": false
		}
	]
}

IntervalUpdate Template:

{
	"topics": [
		{
			"topic": "CumulusMX/IntervalUpdate",
			"data": "{\"time\":\"<#timehhmmss>\",\"press\":<#press rc=y>,\"temp\":<#temp rc=y>,}",
			"retain": false
		}
	]
}

Here is MQTT Explorer:

Screenshot 2023-07-23 133941

I noticed an odd thing, although I don’t know if I’m doing it correctly. When I listen to # on the events, it shows nothing.

I still don’t see any data coming into HA. Any ideas where I go from here?

koying · July 23, 2023, 5:46pm

Your CumulusMX configuration json is irrelevant, here, I’m afraid.

What counts is what you see in mqtt explorer, and the ha error log.
I think you have a wrong trailing comma in your topic values.

petro · July 23, 2023, 5:59pm

data isn’t formatted properly, it seems to be erroring because it’s invalid json. Specifically,

for it to be valid json,

"data": {"time": "<#timehhmmss>", "windspeed": "<#wlatest rc=y>", "winddir": "<#bearing>"},

However, that may be in correct because your topics look mostly correct. What’s messing it up is your trailing comma. That’s not valid json.

I.e. valid

{"x": 1}

not valid

{"x": 1,}

So, assuming that this is valid:

{
	"topics": [
		{
			"topic": "CumulusMX/DataUpdate",
			"data": "{\"time\":\"<#timehhmmss>\",\"windspeed\":<#wlatest rc=y>,\"winddir\":<#bearing>,}",
			"retain": false
		}
	]
}

You want to remove the trailing comma after bearing

{
	"topics": [
		{
			"topic": "CumulusMX/DataUpdate",
			"data": "{\"time\":\"<#timehhmmss>\",\"windspeed\":<#wlatest rc=y>,\"winddir\":<#bearing>}",
			"retain": false
		}
	]
}

sparkytech · July 23, 2023, 8:49pm

Got it finally! Thanks everyone for the guidance, a JSON expert I am not. One more question… the “DataUpdate” works now. The “IntervalUpdate” does not due to the amount of characters in the string. The log shows:

homeassistant.exceptions.InvalidStateError: Invalid state encountered for entity ID: sensor.cumulusmx_interval. State max length is 255 characters.

If I remove many of the values from the IntervalUpdate string, it works fine. I tried splitting this up with the following sensor, but it’s not working (yet).

  - name: "Davis Test Sensor"
    state_topic: "CumulusMX/Interval"
    value_template: "{{ value_json.time }}"
    json_attributes_topic: "CumulusMX/Interval"
    json_attributes_template: >-
      { "New Davis Temp": {{value_json.temp}},
        "New Davis Humidity": {{value_json.humidity}},
        "New Davis Cloudbase": {{value_json.cloudbasevalue}},
        "New Davis Dewpoint": {{value_json.dewpoint}},
        "New Davis Humidex": "{{value_json.humidex}}",
        "New Davis Wind Chill": "{{value_json.wchill}}" }

rafal83 · August 30, 2024, 9:42am

If you are interested

MQTT Discovery for Home assistant

[CumulusMX_HomeAssistant.json](CumulusMX - MQTT discovery for Home Assistant · GitHub)

yestrdave · May 2, 2025, 1:22pm

Im struggling trying to get this working. I have tried everything here and cannot get mgtt explorer to see anything other than the default.