OK, I’ve found out what’s going on. I’m documenting it here so that future people might find this.
If a unit of measurement is defined, then that is used as the measurement name, but entries are tagged with the sensor id. Therefore you need to add a WHERE clause to the query to filter out values. For example:
SELECT mean("value") FROM "%" WHERE ("entity_id" = 'kitchen_humidity') AND $timeFilter GROUP BY time($__interval) fill(previous)
It seems this has been done so you can, for example, average all temperature measurements in the house, although to my experience that doesn’t really work.