My hot water was being weird. I’d get hot water for a while, but then it would get cold. Sounds like cold water getting into the hot loop.
But there’s a check valve. Two check valves in fact. Two spring loaded check valves, installed as the water heater instructions said! And they tested OK. There’s no way it should be cold water getting into the hot loop.
Very odd. Clearly the only reasonable thing to do is to instrument everything.
Finally a use for the otherwise nearly-useless 512kB ESP01 that basically only has one GPIO!
This is in no way novel, so I didn’t make a schematic; it’s just one of those nasty 8-pin ESP01 modules, powered by an AMS1117 LDO with a bunch of a bunch of DS18B20 probes in parallel on GPIO2. They work well enough, powered from the same 3.3v supply as the ESP01, and only need a single 4.7K pull-up.
And what do we see?
Without a doubt, in spite of the check valves, cold water is going the wrong way through the recirculation line. Looking closely right before 23:00:
- the recirculation pump has just finished a warming cycle. Roughly every 20 minutes it circulates water through a loop so that I don’t waste water waiting for the taps to get hot.
- someone starts a shower
- the heater outlet temperature goes hot.
- the heater inlet temperature drops suddenly as water is being drawn from the city supply
- the city inlet temperature decreases as the water that was slightly warmed in the utility closet is used up, and the water is now as cold as underground
- the recirculation line temperature decreases as the warm water that was in the loop is displaced by the cold supply water.
Clearly the check valves had failed, even though they bench-tested OK. 
I’m sure all you devs out there have found modules/libraries where the unit tests pass but the thing doesn’t work in production.
Anyway, with that irrefutable proof, it was off to the hardware store for more check valves, copper pipe, and couplings.
The repair worked. I closed the shut-off valves, drained most of the water, cut out the defective valves, soldered in new ones, and everything worked. That 130 degree temperature spike was from me getting the torch a little too close to the ambient temperature sensor. As a bonus, I could turn the water heater temperature down since I was not longer getting extra cold mixed in at the taps.
Again, nothing novel or difficult here, this is more a story of the kind of repairs you can do for yourself when you have the ability to build your own tools and gather the right data.
From the esphome config…
# Set up 1-Wire bus
one_wire:
- platform: gpio
pin: 2
id: dallas_bus
switch:
- platform: restart
name: "${friendlyname} Restart"
id: ${name}_restart
binary_sensor:
- platform: status
id: ${name}_status
name: "${friendlyname} Status"
text_sensor:
- platform: version
id: ${name}_version
name: "${friendlyname} Version"
sensor:
- platform: uptime
id: ${name}_uptime
name: "${friendlyname} Uptime"
- platform: adc
id: ${name}_adc
pin: VCC
name: "${friendlyname} VCC"
- platform: wifi_signal
id: ${name}_wifi_signal
name: "${friendlyname} WiFi"
- platform: dallas_temp
name: "City Supply"
address: 0x86b68ec335646128
one_wire_id: dallas_bus
update_interval: 5s
- platform: dallas_temp
name: "Heater In"
address: 0xbe39c9c335646128
one_wire_id: dallas_bus
update_interval: 5s
- platform: dallas_temp
name: "Heater Out"
address: 0x642a4ec035646128
one_wire_id: dallas_bus
update_interval: 5s
- platform: dallas_temp
name: "Recirc Out"
address: 0x95592dc335646128
one_wire_id: dallas_bus
update_interval: 5s
- platform: dallas_temp
name: "Ambient"
address: 0x7a920cc335646128
one_wire_id: dallas_bus
update_interval: 5s