HA error for mqtt that does not show up in mqtt explorer

Configuration
1

Hi all, I have HA for 5 years now and really happy it. My main way of integrating IoT around the house is via mqtt. I’d say I’m pretty experienced with the protocol and how to use it in both HA and Node Red.

However, for a while now I get like 3.5k warnings a day in the log for an Erroneous JSON. The reason is, that a too large JSON is sent, it is cut off by HA or the mqtt broker and hence the JSON structure becomes faulty (i.e. prob a bracket is missing because of the cut off).

All ok and well, but for the life of me I cant find the source of the json to reduce its size. I know it comes from an http call to homewizard in node red. Seeing the data it is clearly from there. But where I used to sent the entire response (causing this warning), I now split it in different pieces and sent it to different topics. So this shouldnt occur anymore.

When checking all topics in mqtt explorer, I also dont see the original huge response coming by anymore (as expected). So, my guess is somewhere it still lives in HA and I cant find where it is used. I checked all the related topics in my mqtt configuration in HA and those all have small jsons.

Any idea where to look further?

The warning message in HA is as follows:

Logger: homeassistant.components.mqtt.mixins
Source: components/mqtt/mixins.py:443
integration: MQTT ([documentation](https://www.home-assistant.io/integrations/mqtt), [issues](https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+mqtt%22))
First occurred: October 7, 2024 at 20:59:33 (25403 occurrences)
Last logged: 11:22:29
* Erroneous JSON: {bla bla bla sudden cut off
* Erroneous JSON: {bla bla bla sudden cut off
* Erroneous JSON: {bla bla bla sudden cut off
* Erroneous JSON: Not Found

I also get the following error message that suggests that these are the code pieces involved, since they cant process the JSON

Logger: homeassistant.helpers.template
Source: helpers/template.py:792
First occurred: October 7, 2024 at 20:59:33 (43548 occurrences)
Last logged: 11:22:29
* Error parsing value: 'value_json' is undefined (value: {bla bla bla sudden cut off, template: {{ value_json["values"]["status"] }})
* Error parsing value: 'value_json' is undefined (value: {bla bla bla sudden cut off, template: {{ value_json["values"] | tojson }})
* Error parsing value: 'value_json' is undefined (value: Not Found , template: {{ value_json["values"] | tojson }})
* Error parsing value: 'value_json' is undefined (value: Not Found , template: {{ value_json["values"]["status"] }})

I have searched for all the “value_json[“values”]” pieces in my yaml config files of HA and monitored their mqtt topics and those all have the new smaller JSONs. Are there other places I could be reading mqtt directly which causes me to miss a topic?

I’m lost and I don’t like the frequency of this warning. I want a clean config so would really like to address this.

Thanks for any help!!

edit: The json mentioned in HA is 1100 characters long. So I have made a node red flow that checks the length over every mqtt message and I only see messages longer than 1000 characters for frigate, double take and rpi-monitor. This suggests even more so that there is cache of the original long json in HA somewhere. My install is quite old, so maybe somwhere along the way something got a not-that-great update causing this. But I’m so curious where. I always update to latest versions of HA within a month or two btw.

2

Nope, it will always be getting information from topics.

Yes, you should look at all topics provided by anything that is automatically discovered in the MQTT integration.

3

Neither HA nor the broker will cut off the JSON - however it will be cut off if you store it in state and later expect to use it as JSON. State can only be 255 characters, so this can be truncated. It may also be that the “Erroneous JSON” error just truncates the message given it could be any length.

Maybe post the actual JSON from Explorer plus the sensor definitions of whatever is listening to it.

Other than that, I’d be looking at the entities you’re creating from that JSON - for example if you’re using json_attributes_template on an MQTT sensor and your template does not return a dictionary, I believe you will get this error.

4

I have checked all the topics by making a node-red flow that subscribed to all topics and filtering for long messages. It does not show up. So it must be something in HA. @michaelblight suggested a template sensor might have it stored, which definitely could be the case. I’ll investigate.

5

I’m triggered by your “stored in state” mention. I think I’ve stored it in a state in the past but dont anymore. It could still be in some sensors state and give the errors while I cant find it anymore in my config. Will have a look.

6

Hmm, I’ve queried the state table in the database and there is nothing with one of the unique words of the json in either the state or attribute column. I used a very inclusive like statement:

select
*
from states s 
where s."attributes" like '%GardenaO%'
limit 3
;

I’ve also included my node-red flow to check all mqtt topics. Just in case I missed something there, though I wouldnt know how it could miss anything.

Screenshot 2024-10-21 at 10.00.57
Screenshot 2024-10-21 at 10.00.571920×640 94.2 KB

Screenshot 2024-10-21 at 10.01.23
Screenshot 2024-10-21 at 10.01.231158×608 45 KB

The call in node red to homewizard is a REST call. I looked for a rest sensor in my config and found an old one commented out, but the sensor name is not found anymore in the entities.

7

Just now spotted a datetime in the JSON that updates. So there are fresh messages coming in somehow. I’m so puzzled haha.

8

That’s probably from an old database schema. These days I think you’d need to look in the “state_attributes” table.

9

Could well be, unfortunately nothing in state_attributes too…

10

I’m not sure how you got down that rabbit hole, this is only going to be related to MQTT and MQTT discovery.

Start by searching your MQTT configuration for these templates

Template is the portion between the {{' and }}`.

If you don’t have that in your config, these errors are coming from MQTT discovery, in which those templates may exist in your MQTT topics, assuming that the discovery topics are retained.

Then once you figure that out, you can look at the discovery info to see what topic that config is creating entities from. Its likely that those devices are outputting an empty value every now and then which results in that error.

11

Thanks, will look into that and log those topics

12

Ok, I found indeed that sometimes an empty message gets posted on these topics which leads to the errors. This is hard to see in mqtt explorer so I’m glad you suggested this. I perform an http request in node red and sometimes the call failed and json parsing then also failed, but still produced a message (though empty).

Still really weird to see the huge json in the error message in home assistant. But I was thinking… obviously HA has warning message aggregation and maybe this warning happened first a long time ago when I still sent long messages and it still shows that message. Dunno, but would explain why I went in the rabbit hole.

I fixed the empty message generation 45 minutes ago and all warnings disappeared.

Thanks thanks thanks!

1 Like