Node-Red Tesseract OCR to read heatpump temperature

smi · January 10, 2022, 3:42pm

Hi,

since my heatpump does not provide any API and I was not able to reverse engineer the display / controller yet, I’d like to read the current temperature from the display using OCR.

For this, I added node-red-contrib-tesseract to Node-Red.
Unfortunately, it only reads out “SET TEMP” and some characters / symbols.

Any idea on how to get the temperature reading?

Here is my flow:

[{"id":"7ebf52847bf252b1","type":"tab","label":"OCR Water Heatpump Reading","disabled":false,"info":"","env":[]},{"id":"1e98a117305d51df","type":"tesseract","z":"7ebf52847bf252b1","name":"","language":"eng","x":800,"y":500,"wires":[["141115f7f34bd410","e7573fa98a72f8cb"]]},{"id":"141115f7f34bd410","type":"debug","z":"7ebf52847bf252b1","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1030,"y":500,"wires":[]},{"id":"1cbe5ed2aedb7dbf","type":"inject","z":"7ebf52847bf252b1","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":410,"y":500,"wires":[["fb1ce425175bab2c"]]},{"id":"fb1ce425175bab2c","type":"www-request","z":"7ebf52847bf252b1","name":"","method":"GET","ret":"bin","url":"https://www.online360.at/heatpump4.jpeg","follow-redirects":true,"persistent-http":true,"tls":"","x":590,"y":520,"wires":[["1e98a117305d51df"]]},{"id":"e7573fa98a72f8cb","type":"debug","z":"7ebf52847bf252b1","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"tesseract","targetType":"msg","statusVal":"","statusType":"auto","x":1050,"y":580,"wires":[]}]

Thanks!

Mikefila · January 10, 2022, 6:01pm

How are you getting the image of the screen. I’m thinking that this has more to do with the conversion software/image quality than nodered itself. If you send an image with clean text and numbers does it work correctly?

smi · January 10, 2022, 10:00pm

Here is an example that I tried: https://www.online360.at/heatpump4.jpeg

Quality is good but I think the gaps between each element of a digit might be a problem.

smi · January 10, 2022, 10:18pm

According to reddit an Co. also others seem to face issues with tesseract and LCDs: Text detection on Seven Segment Display via Tesseract OCR - Stack Overflow

I’ll try to get some trained data into the node red plugin / try any other solutions.

Mikefila · January 10, 2022, 11:21pm

I just tried that image in a few online ocr websites and none detected text. I think there is too much color/contrast and the bleed is maybe whats causing it. If you can turn it down it may help.

smi · January 11, 2022, 11:16am

I played around with some image pre-processing and the image itself looks pretty good after that:

[{"id":"7ebf52847bf252b1","type":"tab","label":"OCR Water Heatpump Reading","disabled":false,"info":"","env":[]},{"id":"1e98a117305d51df","type":"tesseract","z":"7ebf52847bf252b1","name":"","language":"eng","x":2120,"y":520,"wires":[["141115f7f34bd410","e7573fa98a72f8cb"]]},{"id":"141115f7f34bd410","type":"debug","z":"7ebf52847bf252b1","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":2290,"y":520,"wires":[]},{"id":"1cbe5ed2aedb7dbf","type":"inject","z":"7ebf52847bf252b1","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":90,"y":500,"wires":[["94c01b2ce03a08b4"]]},{"id":"e7573fa98a72f8cb","type":"debug","z":"7ebf52847bf252b1","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"tesseract","targetType":"msg","statusVal":"","statusType":"auto","x":2310,"y":600,"wires":[]},{"id":"0aa5d1b2b0cb1d67","type":"jimp-image","z":"7ebf52847bf252b1","name":"","data":"payload","dataType":"msg","ret":"img","parameter1":"529","parameter1Type":"num","parameter2":"620","parameter2Type":"num","parameter3":"433","parameter3Type":"num","parameter4":"375","parameter4Type":"num","parameter5":"","parameter5Type":"msg","parameter6":"","parameter6Type":"msg","parameter7":"","parameter7Type":"msg","parameter8":"","parameter8Type":"msg","sendProperty":"payload","sendPropertyType":"msg","parameterCount":4,"jimpFunction":"crop","selectedJimpFunction":{"name":"crop","fn":"crop","description":"crop to the given region","parameters":[{"name":"x","type":"num","required":true,"hint":"the x coordinate to crop form"},{"name":"y","type":"num","required":true,"hint":"the y coordinate to crop form"},{"name":"w","type":"num","required":true,"hint":"the width of the crop region"},{"name":"h","type":"num","required":true,"hint":"the height of the crop region"}]},"x":510,"y":500,"wires":[["60aa4999b5e03163","741d022c0556b539"]]},{"id":"3061ecab5bc47659","type":"jimp-image","z":"7ebf52847bf252b1","d":true,"name":"","data":"payload","dataType":"msg","ret":"img","parameter1":"10","parameter1Type":"num","parameter2":"","parameter2Type":"msg","parameter3":"","parameter3Type":"msg","parameter4":"","parameter4Type":"msg","parameter5":"","parameter5Type":"msg","parameter6":"","parameter6Type":"msg","parameter7":"","parameter7Type":"msg","parameter8":"","parameter8Type":"msg","sendProperty":"payload","sendPropertyType":"msg","parameterCount":1,"jimpFunction":"quality","selectedJimpFunction":{"name":"quality","fn":"quality","description":"Set the quality of the image. Useful for reducing size of image before calling the write function.","parameters":[{"name":"quality","type":"num","required":true,"hint":"Quality value 1 ~ 100"}]},"x":970,"y":500,"wires":[["1ac2a88b62b31ffc","a69c293f8943708c"]]},{"id":"1ac2a88b62b31ffc","type":"image viewer","z":"7ebf52847bf252b1","name":"","width":160,"data":"payload","dataType":"msg","active":true,"x":1070,"y":620,"wires":[[]]},{"id":"94c01b2ce03a08b4","type":"jimp-image","z":"7ebf52847bf252b1","name":"","data":"https://www.online360.at/heatpump4.jpeg","dataType":"str","ret":"img","parameter1":"","parameter1Type":"msg","parameter2":"","parameter2Type":"msg","parameter3":"","parameter3Type":"msg","parameter4":"","parameter4Type":"msg","parameter5":"","parameter5Type":"msg","parameter6":"","parameter6Type":"msg","parameter7":"","parameter7Type":"msg","parameter8":"","parameter8Type":"msg","sendProperty":"payload","sendPropertyType":"msg","parameterCount":0,"jimpFunction":"none","selectedJimpFunction":{"name":"none","fn":"none","description":"Just loads the image.","parameters":[]},"x":260,"y":500,"wires":[["82877f4d59ee2a6f","0aa5d1b2b0cb1d67"]]},{"id":"82877f4d59ee2a6f","type":"image viewer","z":"7ebf52847bf252b1","name":"","width":160,"data":"payload","dataType":"msg","active":true,"x":410,"y":640,"wires":[[]]},{"id":"60aa4999b5e03163","type":"image viewer","z":"7ebf52847bf252b1","name":"","width":160,"data":"payload","dataType":"msg","active":true,"x":590,"y":640,"wires":[[]]},{"id":"a69c293f8943708c","type":"jimp-image","z":"7ebf52847bf252b1","name":"","data":"payload","dataType":"msg","ret":"img","parameter1":"","parameter1Type":"msg","parameter2":"","parameter2Type":"msg","parameter3":"","parameter3Type":"msg","parameter4":"","parameter4Type":"msg","parameter5":"","parameter5Type":"msg","parameter6":"","parameter6Type":"msg","parameter7":"","parameter7Type":"msg","parameter8":"","parameter8Type":"msg","sendProperty":"payload","sendPropertyType":"msg","parameterCount":0,"jimpFunction":"greyscale","selectedJimpFunction":{"name":"greyscale","fn":"greyscale","description":"remove colour from the image","parameters":[]},"x":1150,"y":500,"wires":[["ef2631aa6d5b6259","d85da69001ecc8cd"]]},{"id":"ef2631aa6d5b6259","type":"image viewer","z":"7ebf52847bf252b1","name":"","width":160,"data":"payload","dataType":"msg","active":true,"x":1240,"y":620,"wires":[[]]},{"id":"d85da69001ecc8cd","type":"jimp-image","z":"7ebf52847bf252b1","name":"","data":"payload","dataType":"msg","ret":"img","parameter1":"-0.5","parameter1Type":"num","parameter2":"5","parameter2Type":"msg","parameter3":"","parameter3Type":"msg","parameter4":"","parameter4Type":"msg","parameter5":"","parameter5Type":"msg","parameter6":"","parameter6Type":"msg","parameter7":"","parameter7Type":"msg","parameter8":"","parameter8Type":"msg","sendProperty":"payload","sendPropertyType":"msg","parameterCount":0,"jimpFunction":"invert","selectedJimpFunction":{"name":"invert","fn":"invert","description":"invert the image colours","parameters":[]},"x":1320,"y":500,"wires":[["e8b4da2d7f49322c","ef08ecd3a78ef975"]]},{"id":"e8b4da2d7f49322c","type":"image viewer","z":"7ebf52847bf252b1","name":"","width":160,"data":"payload","dataType":"msg","active":true,"x":1420,"y":620,"wires":[[]]},{"id":"94dd8a2e37268318","type":"jimp-image","z":"7ebf52847bf252b1","name":"","data":"payload","dataType":"msg","ret":"buf","parameter1":"","parameter1Type":"msg","parameter2":"","parameter2Type":"msg","parameter3":"","parameter3Type":"msg","parameter4":"","parameter4Type":"msg","parameter5":"","parameter5Type":"msg","parameter6":"","parameter6Type":"msg","parameter7":"","parameter7Type":"msg","parameter8":"","parameter8Type":"msg","sendProperty":"payload","sendPropertyType":"msg","parameterCount":0,"jimpFunction":"none","selectedJimpFunction":{"name":"none","fn":"none","description":"Just loads the image.","parameters":[]},"x":1990,"y":520,"wires":[["1e98a117305d51df"]]},{"id":"ef08ecd3a78ef975","type":"jimp-image","z":"7ebf52847bf252b1","name":"","data":"payload","dataType":"msg","ret":"img","parameter1":"15","parameter1Type":"num","parameter2":"","parameter2Type":"msg","parameter3":"","parameter3Type":"msg","parameter4":"","parameter4Type":"msg","parameter5":"","parameter5Type":"msg","parameter6":"","parameter6Type":"msg","parameter7":"","parameter7Type":"msg","parameter8":"","parameter8Type":"msg","sendProperty":"payload","sendPropertyType":"msg","parameterCount":0,"jimpFunction":"normalize","selectedJimpFunction":{"name":"normalize","fn":"normalize","description":"normalize the channels in an image","parameters":[]},"x":1490,"y":500,"wires":[["a0254699369ec9e0","5d35977b376d7f11"]]},{"id":"a0254699369ec9e0","type":"image viewer","z":"7ebf52847bf252b1","name":"","width":160,"data":"payload","dataType":"msg","active":true,"x":1620,"y":620,"wires":[[]]},{"id":"5d35977b376d7f11","type":"jimp-image","z":"7ebf52847bf252b1","name":"","data":"payload","dataType":"msg","ret":"img","parameter1":"165","parameter1Type":"num","parameter2":"","parameter2Type":"none","parameter3":"","parameter3Type":"msg","parameter4":"","parameter4Type":"msg","parameter5":"","parameter5Type":"msg","parameter6":"","parameter6Type":"msg","parameter7":"","parameter7Type":"msg","parameter8":"","parameter8Type":"msg","sendProperty":"payload","sendPropertyType":"msg","parameterCount":3,"jimpFunction":"threshold","selectedJimpFunction":{"name":"threshold","fn":"threshold","description":"apply one or more functions","parameters":[{"name":"max","group":"options","type":"num","required":true,"hint":"max value of byte 0 ~ 255"},{"name":"replace","group":"options","type":"num","required":false,"hint":"replace with byte 0 ~ 255. Default is 255"},{"name":"autoGreyscale","group":"options","type":"bool","required":false,"hint":"default is true"}]},"x":1680,"y":500,"wires":[["047e605c45c44ffc","9b7bb6387eba49a2"]]},{"id":"047e605c45c44ffc","type":"image viewer","z":"7ebf52847bf252b1","name":"","width":160,"data":"payload","dataType":"msg","active":true,"x":1820,"y":620,"wires":[[]]},{"id":"741d022c0556b539","type":"jimp-image","z":"7ebf52847bf252b1","name":"","data":"payload","dataType":"msg","ret":"img","parameter1":"","parameter1Type":"auto","parameter2":"50","parameter2Type":"num","parameter3":"RESIZE_BEZIER","parameter3Type":"resizeMode","parameter4":"","parameter4Type":"msg","parameter5":"","parameter5Type":"msg","parameter6":"","parameter6Type":"msg","parameter7":"","parameter7Type":"msg","parameter8":"","parameter8Type":"msg","sendProperty":"payload","sendPropertyType":"msg","parameterCount":3,"jimpFunction":"resize","selectedJimpFunction":{"name":"resize","fn":"resize","description":"resize the image. One of the w or h parameters can be set to automatic (\"Jimp.AUTO\" or -1).","parameters":[{"name":"w","type":"num|auto","required":true,"hint":"the width to resize the image to (or \"Jimp.AUTO\" or -1)"},{"name":"h","type":"num|auto","required":true,"hint":"the height to resize the image to (or \"Jimp.AUTO\" or -1)"},{"name":"mode","type":"resizeMode","required":false,"hint":"a scaling method (e.g. Jimp.RESIZE_BEZIER)"}]},"x":710,"y":500,"wires":[["3061ecab5bc47659","a69c293f8943708c"]]},{"id":"9b7bb6387eba49a2","type":"jimp-image","z":"7ebf52847bf252b1","name":"","data":"payload","dataType":"msg","ret":"img","parameter1":"1","parameter1Type":"num","parameter2":"","parameter2Type":"msg","parameter3":"","parameter3Type":"msg","parameter4":"","parameter4Type":"msg","parameter5":"","parameter5Type":"msg","parameter6":"","parameter6Type":"msg","parameter7":"","parameter7Type":"msg","parameter8":"","parameter8Type":"msg","sendProperty":"payload","sendPropertyType":"msg","parameterCount":1,"jimpFunction":"blur","selectedJimpFunction":{"name":"blur","fn":"blur","description":"fast blur the image by r pixels","parameters":[{"name":"r","type":"num","required":true,"hint":"the pixel radius of the blur"}]},"x":1840,"y":500,"wires":[["64b532e612f983ba","94dd8a2e37268318"]]},{"id":"64b532e612f983ba","type":"image viewer","z":"7ebf52847bf252b1","name":"","width":160,"data":"payload","dataType":"msg","active":true,"x":2000,"y":620,"wires":[[]]}]

There is some trained data on seven segment displays but I have no idea on how to get these into the node red node (or is it called module or so?)

Another option I tried was the following integration: Seven Segments OCR - Home Assistant
But hass.io does not know the command “ssocr” even the integration is installed and therefore it’s hard to test.

smi · January 12, 2022, 8:57am

Ok so the solution is now as follows:

Pre-Processing the image using Node-Red to have a clean black/white image.
Then SSOCR does the magic to read out the value.

This is working with a local image for testing and I will present the whole flow when everything is up and running including the new webcam (ESP32 cam)