I have a sensor from an image processing which flickers a little bit causing the recorder database to be quite large and cause lovelace to freeze or be very slow when trying to look into the sensor records. Though I have fixed the sensor problem, it got me to look into how to improve/optimize performance of the data processing of home assistant.
I will share here something new which I did not find in the forum.
I of course first moved the recorder database from the default sqlite to mariadb but my next step was to look at the json encoding/decoding library. I ended up switching home assistant to orjson from the default python json library. The performance gain is documented here:
There was an attempt to do this in the main home assistant here but was later reverted here. The gain is documented here, published by testing for rapidjson, yet another json library I had considered. You can see than the decoding only improves by 30% by the encoding in average improves 8x. The latter is what is making the switch worth the while.
I noted also that the code structure of home-assistant with the encoding helper made the switch easy for the encoding function but not for decoding. I had to go to each component and make the switch in the code component by component.