New Integration performance - 100s of entity updates every second

Development
1

I have developed a custom integration that updates 100s of entities per second. I am looking at ways to reduce the negative performance effect on HA as it is lagging a lot.
I have excluded the domain from the logbook but I need the created entities to have a history.
What else can I do?
The knock on effect of recorder, history and logbook I think is quite significant.

2

I guess you have a special use case, but why?

3

Can you tell us a little more about the integration? What type of entities? Are they being polled or waiting on state changes (listening for events)? What hardware is being used?

4

In general the integration dynamically creates and updates entities based on binary data received from a Can Bus.
In ESPHome via an esp32 I update a general entity with lambda c++ code
id(canbus_frame).publish_state(combinedHex);
This is connected to a Can Bus that carries packets of binary data.
And then within a custom integration I catch the sensor_state_change for that entity updated from esphome (canbus_frame) and process it into individual entities via a class that inherits from the Entity class. This is used to create and update the entity.
I have polling off for these dynamically created entities.
I am using a Rpi 4 - 8GB.
That does help?
Overall it works very well. I am using async for the majority of the integration.

5

In general the integration dynamically creates and updates entities based on binary data received from a Can Bus.
Maybe I am pushing the limits of what HA can achieve on a Rpi 4

6

Are you writing to a SD card or SSD?

7

I have tried both SSD and SD card. Yes it is faster on a SSD but I am writing an integration for others to use so it needs to work with an SD card.

8

This may be an option:

9

Is that really necessary or isn’t it enough to update every x seconds instead.
Such amount of state writes I wonder if it makes sense for anyone.

10

Well that is how much data is coming in. I have configured some throttling so that it skips some of the data. It seems to help.

11

Thanks. I have tried most of these.
Switching off the logbook helped a little.