0.98 Bug? MQTT Temperature in F is converted again C->F

[Linwood (Linwood Ferguson)]

I’ve been experimenting with ESPHOME and had a working prototype just sitting and sending temp, humidity and soil reports via MQTT.

Installed 0.98 and temperature almost doubled. I realized it exactly went up by the C->F conversion. No change was made to the esphome device.

The ESPHOME code converts from C (native in the sensor it creates) to F as follows (this is in the firmware YAML not HA):

sensor:
  - platform: dht
    pin: D2
    temperature:
      name: "Test Temperature"
      filters:
      - lambda: return x * (9.0/5.0) + 32;
      unit_of_measurement: '°F'

That is working as reported by both its logs, and as seen here from mqtexplorer.

[0;36m[D][sensor:092]: 'Test Temperature': Sending state 75.74001 °F with 1 decimals of accuracy

{  
   "unit_of_measurement":"°F",
   "icon":"mdi:thermometer",
   "name":"Test Temperature",
   "state_topic":"testhumidity/sensor/test_temperature/state",
   "availability_topic":"testhumidity/status",
   "unique_id":"ESPsensortest_temperature",
   "device":{  
      "identifiers":"840d8eaa278f",
      "name":"testhumidity",
      "sw_version":"esphome v1.13.6 Aug 28 2019, 13:21:19",
      "model":"PLATFORMIO_NODEMCUV2",
      "manufacturer":"espressif"
   }
}

The Home Assistant sensor has no configuration, it is auto-discovered, and appears as follows in states:

[sensor.test_temperature]
(http://ha:8123/developer-tools/state#) 
167.7 unit_of_measurement: °F 
friendly_name: Test Temperature 
icon: mdi:thermometer

(Those two quotes are from slightly different times so don’t be surprised that there’s a few tenth’s off on the conversion).

It seems as though with 0.98 (or at least after the install if not because of the version) it began to do an implicit conversion even though there’s nothing in configuration.yaml to do so.

Is there some new presumption of unit in MQTT that is now being converted?

[tom_l]

Have a read of this:

Temperature just switched from C to F - why?

temperature units are global in home assistant. If you specify a sensor with a given unit, home assistant will convert it to the global unit in the user interface. To change your global unit, see here:

and this:

Why is unit_of_measurement: '°F' not working?

How have you configured unit_system? The choices are metric or imperial. If you set it to imperial, Home Assistant will assume all received temperature values are in Fahrenheit and will display them in Fahrenheit. homeassistant: unit_system: imperial If you have unit_system: metric and wish to display a temperature sensor’s value in Fahrenheit then unit_of_measurement: '°F' will cause the value to be converted from metric to imperial.

[Linwood (Linwood Ferguson)]

The units for the system are imperial. I have a dozen or so zwave sensors that work fine in that regard. The HA install is not new, only this mqtt sensor is newly done.

It ran for about a day and registered correctly, only after the reboot for the 0.98 install did it double convert. Maybe it was the reboot.

As you can see MQTT is sending the unit of measurement as ‘°F’. It seems as though HA is taking that, assuming it is imperial, and converting it to F despite already being there.

I should note that the original data sent many hours earlier was not F, and not sure if it was labeled or not (whatever the defaults in esphome). When I saw it coming as C I converted in esphome, labeled it in esphome, updated the firmware and got F in HA, got the right value in HA, all was good.

Then later that day (or maybe the next) upgraded to 0.98 and that’s when it went from 80f to 176F.

Could something from the initial data be “stuck” and only seen by HA on reboot (or newly by 0.98)?

PS. I did verify the time of the sudden change corresponds to the install. I am also sure I restarted (not rebooted) HA several times during the day without it changing temperatures. The only reboot was for the 0.98 install.

[petro]

delete your database, you borked it making the change on esphome side and letting it stick into the database.

EDIT: Now that I think about it a little more. It should only affect your symbol in the UI. Deleting the database will fix that. But the number should not be double converted.

[Linwood (Linwood Ferguson)]

I didn’t want to delete the whole database as I have a lot of history there. I did delete all rows in ‘states’ and ‘events’ that contained rows for that entity. No change.

I’m also trying to avoid restarting ha right now, as I’m also trying to build a zwave map and still have a couple sensors that have not reported end, which is handicapping me.

Where are the units of an entity stored? I.e. the ones used to decide if to do a conversion?

PS. I tried deleting the entity from the entity registry without impact. It looks like I need to remove the entire integration to remove it for real. I think I’m going to change its name and re-flash so it gets a whole new entity discovered and see what happens.

[Linwood (Linwood Ferguson)]

I changed the yaml in esphome to change the name, and re-flashed OTA. Interestingly, without a restart on HA, it removed the old sensor and added the new name.

This one is showing the right temperature with no (extra) conversion.

This is with no change in what the sensor is sending except the name.

So indeed, it would appear to be something stashed somewhere from the initial setup that does not change when I add a unit-of-measurement.

The messages via MQTT are identical except the name.

So it’s data somehow, somewhere stashed in HA.

But it’s data that did not come forward until I did an install of 0.98 – whether due to something about 0.98, or something about the install/reboot process that caused it to surface.

[Linwood (Linwood Ferguson)]

One more set of observation: I changed the name back to the original and reflashed. Once it started sending, HA matched up three sensors (humidity and soil moisture) with what was there before, but the temperature was discovered as sensor.test_temperature_2. Clearly it found a name collision but decided it was not the same and created a new one, and this one has the right units.

In the mqtt database the names matched, and old data is still present (i.e. it lumped it all together), so it was definitely HA that decided to separate to a test_temperature_2.

And it initially started sending correct temperatures (i.e. only the conversion in the esp8266).

Then, unrelated, was testing a generator getting ready for the storm, and my VM rebooted, which rebooted HA. After rebooting, the test_temperature_2 now was doing a double convert and about 170+ degrees.

Changed back to the new device name, and this time got a test_temperature_2 under the new device (for reasons unclear). And weirdly here it said “unavailable”. I deleted the whole mqtt integration, restarted HA, let it rediscover.

So I kept exploring and found I wasn’t getting correct discovery without the API in the esphome yaml. Not sure why, I thought that was for non-mqtt (this yaml is running on a whole different HA instance). Put it back, now started seeing the new sensor names, but still getting oddness.

Here’s an example. I’m actually getting the same sensor twice. When I look in mqtt explorer I see only one with that name. I have no idea why I’m seeing it twice in HA. Under developer tools, states there’s only one (without the extra decimal digits).

I looked in states, and here is the entire set of data in states, that corresponds roughly to the time of the graph shown (the data is a bit longer than the graph, but starts at the beginning). The 33.8 temps are a side effect of the device reporting 1C immediately after flashing. I flashed again about 1:58 so you see it twice in the data and in the graph. But only the top graph.

The bottom graph is… different. It’s about right for the data and time frame. But seems to exclude some of the data points that converted from 1c.

entity_id                 state        attributes                                                                                           last_updated
------------------------  -----------  ---------------------------------------------------------------------------------------------------  --------------------------
sensor.test2_temperature  unavailable  {"unit_of_measurement": "\u00b0C", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 17:46:05.677485
sensor.test2_temperature  unknown      {"unit_of_measurement": "\u00b0C", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 17:46:05.712339
sensor.test2_temperature  33.8         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 17:46:07.607957
sensor.test2_temperature  unavailable  {"unit_of_measurement": "\u00b0C", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 17:47:22.350852
sensor.test2_temperature  33.8         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 17:47:22.477755
sensor.test2_temperature  80.4         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 17:47:25.945021
sensor.test2_temperature  unavailable  {"unit_of_measurement": "\u00b0C", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 17:56:29.760103
sensor.test2_temperature  80.6         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 17:56:29.778625
sensor.test2_temperature  33.8         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 17:56:33.309839
sensor.test2_temperature  unavailable  {"unit_of_measurement": "\u00b0C", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 17:59:59.810984
sensor.test2_temperature  33.8         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 17:59:59.817962
sensor.test2_temperature  80.8         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:00:03.744866
sensor.test2_temperature  81.0         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:00:33.783447
sensor.test2_temperature  80.8         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:00:48.800983
sensor.test2_temperature  81.0         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:01:18.778602
sensor.test2_temperature  80.8         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:01:33.805346
sensor.test2_temperature  81.0         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:01:48.791092
sensor.test2_temperature  81.1         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:02:48.801180
sensor.test2_temperature  81.0         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:03:33.791581
sensor.test2_temperature  81.1         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:03:48.792408
sensor.test2_temperature  81.0         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:04:48.780504
sensor.test2_temperature  81.1         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:05:18.791504
sensor.test2_temperature  81.0         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:06:03.783631
sensor.test2_temperature  81.1         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:06:33.792941
sensor.test2_temperature  81.0         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:06:48.791981
sensor.test2_temperature  81.1         {"unit_of_measurement": "\u00b0F", "friendly_name": "Test2 Temperature", "icon": "mdi:thermometer"}  2019-08-29 18:09:03.790878

So what does all this mean – why am I getting two graphs? That’s usually due to different units of measurements, but here the unit of measurement is being inferred by HA. MQTT is getting degrees C entirely, and the esphome yaml has no units of measurement at all for temperature. But you can see it sometimes showing C and sometimes F, based on availability.

Which in turn is screwing up the graph?

There’s something here that is simply not working right.

[Linwood (Linwood Ferguson)]

So I am not certain, but I really think it’s a 0.98 thing.

The rPi I have esphome running on is obviously running ha, and I realized it is on 0.97.2.

So I copied in the mqtt section from the 0.98 configuration.yml and put it in this one, so both are drawing the same data from the same mqtt database.

The chart on the 0.97.2 looks exactly as expected – one line, it has those odd drops to 33f but that’s reflected in the data. There’s no second chart with extra decimal points.

I think 0.98 is doing something weird in the conversion. The extra decimals in the graph, the fact there are two graphs for one item… something.

Going to upgrade 0.97.2 to 0.98 and see what happens. If both then go wonky, that seems pretty clear.

[petro]

If you end up finding something odd, make sure to write up an issue with all this information on github.

[Linwood (Linwood Ferguson)]

Well, another beautiful theory ruined by an ugly fact.

In the rPi that I upgraded to 0.98, the sensor is working just fine.

There’s something corrupt inside my other HA instance, the real one. Not quite sure what to do about it, but at least at this point it does not appear directly related to 0.98, at least not repeatably so.

I’m not sure where to go. I’ll probably have another go at finding and deleting all database entries I can find and try to start clean. Maybe rebuild the esphome configuration with a fresh config file, not just change the name.

[Linwood (Linwood Ferguson)]

Well, I stand corrected. A bit later it started doing this:

I have no idea what’s happening.

I also have no idea why the device keeps reverting occasionally to sending 1 for temp and 2 for humidity, then I reboot and it works fine. I think that’s a separate problem, but maybe the bogus temps are part of the issue. Though both numbers are valid numbers.