Hi!
I solved it - as good as I can. But I’m not feeling good with my solution so I request for comments.
As HA newbie, I’m working on a solution but by going further step by step I often end up with the feeling I’m somehow misusing HA concepts. Then researching and changing my approach, but I’m still not satisfied. Let me describe the steps I took, the solutions and obstacles I found and hopefully we can find some experts here giving the right hints. 
My personal goal is to measure electric power related stuff in the context of a private house with grid, PV/battery, eV and dedicated power consumers, especially those that are shapable like dishwasher, washing mashine and dryer. So pretty standard I’d say, except the fact that my topology and measurement idea doesn’t meet the standard HA energy dashboard.
The problem:
We want to measure different variables and we want to derive variables from that.
One simple example is creating a sum (e.g. meter_A + meter_B = sum), but what also is required is a difference (meter_Sum - meter_A = B). In my case I have both:
- sum_metered := meter_dryer + meter_HVAC + meter_whatever …
- rest = meter_Sum - sum_metered
The obstacles:
- Derived values are meaningless if they are computed out of changing and unsychronized sources
- Dervied entities are computed to often, because in HA standard behavior the are updated after each update of their source entities. If n ist the number of sources, this leads to n-1 wrong extra values between two valid values
Typical erroneous effect is wrong impulses:
The theory:
We cannot absolutely synchronize our measurements. Each meter sits on a different location and each digitizing meter decides for certain time intervals for sampling the truth. So if we’d like to overcome this, we would need to estimate our jitter, low pass our signals, choose appropriate sample frequencies and synchronize our meters accordingly. We would end up in graphs that are smothed so much that they may become more useless than the ones we have. Example: Wallbox pulls up from 6W to 11kW with <<1 sec while our graph could need a minute to reflect that edge in a smooth raising curve.
The optimization:
But! By doing straight forward approach, i.e. configuring metered entities and helper entities for computation, I get the errors shown in the picture above at nearly every edge. Knowing that this behavior can’t be avoided completely, we at least want to try that we generate this meaningless values at nearly no edge.
Measure 1: Minimizing meter response time
In my case, I have a bunch of RS485 MODBUS meters. Regarding the jitter: shit in gives shit out, so I did the following measures:
- I read out as many registers as possible in one MODBUS read (using “virtual_count” of modbus sensors) and
- I increased the RS485 baud rate to the maximum I can do in my setup so I have the fastest communication I can get with my setup (using proper terminations etc.) to minimize jitter from that cause.
Measure 2: Synchronizing and optimizing physical sensor updates
In MODBUS integration, according to the HA documentation, each sensor needs to have a scan_interval.
Obstacle 1: scan_intervals are not synchronized
As reported by the TO, when I configure scan interval x, HA/modbus integration does not align and syncrhonize these intervals, so it’s somewhat random and don’t know maybe its also drifting
Workaround: I set the scan_interval to 20 years and then triggered the entity updates by automations. I found out that HA performs entity updates sequentially, even if they are triggered by one action. So used that to avoid conflicts on the RS485 modbus proxy and created one automation for each rs485 bus. So the busses can run concurrently, but for each bus there is one action to update all entities of this bus sequentially. As trigger I use time_pattern to tigger updates of all busses synchronously (e.g. with ‘minutes: “/5”’).
Now we have a system clock!
Note: I found out that I cannot avoid triggering the update of modbus sensors when HA starts up. This happens always. The next update will be either when the automation is executed or when the uptime reaches a multiple of 20 years. 
Strange effect here:
By increasing the baud rate, i succeeded in getting the values from each modbus in <<1 second. I was able to set the automation trigger to “every second” and I got real nice real time graphs. Now only a couple of days later, and for some reasons I don’t know, the same automations run >2 seconds.
Measure 3: Using sensor templates for calculated values
For summing up the (now synchronized) meter variables, I first tried to use sensor templates (configuration.yaml/template/sensor). I found out that even if I update the source entities of such a derived entity within one automation action (see measure 2), HA updates the derived entity multiple times. Good job, HA, respect, but not want we want here. Because all intermediate values are wrong. Only the last update per clock cycle provides correct results.
Obstacle 2: triggers of template sensors cannot be influenced
I found no possibility to get rid of these unwanted triggers with the usage of template sensors.
Measure 4: Using trigger templates for calculated values
I changed configuration.yaml/template/sensor to configuration.yaml/template/trigger with a linked sensor.
By default this entity now behaves like the template sensor, i.e. it is triggered after every update of a source entity.
Idea “Condition”:
I think about sticking to this event based approach, but as of now I have no idea how to filter the “wrong” events and wait for the “right” one - the last one of a clock cycle. Maybe it’s an idea to add a condition to the trigger that looks at the timestamps of the source entities, and if they are << epsilon.
Measure 5: Using time_patters for creating time slices
As of now I opted for a statically scheduling approach with time slices. I use time_pattern to fire these updates 3 seconds after the begin of each clock cycle, assuming that all modbus sensor updates are now done in <<3 seconds. I have also entities that derive from derived entites (see above), so I scheduled those to 4 seconds after begin of clock cycle.
Now measure 5 is my status quo and it comes with some issues and obstacles. I currently use a clock cycle of 5 seconds. Let me know if you think that’s insane. But I have to learn my consumers, and maybe in a couple of months I can sample this down to 5 minutes. Also I’m monitoring load, because I’m reaching power limits. So for such a phase it’s good that HA can stand these frequencies, at least it seems to do so. Let me know if this causes issues.
Obstacle 2: Time patterns can get long and need to be repeated in yaml very often.
My template trigger looks like this:
- trigger:
- trigger: time_pattern
seconds: 3
- trigger: time_pattern
seconds: 8
- trigger: time_pattern
seconds: 13
- trigger: time_pattern
seconds: 18
- trigger: time_pattern
seconds: 23
- trigger: time_pattern
seconds: 28
- trigger: time_pattern
seconds: 33
- trigger: time_pattern
seconds: 38
- trigger: time_pattern
seconds: 43
- trigger: time_pattern
seconds: 48
- trigger: time_pattern
seconds: 53
- trigger: time_pattern
seconds: 58
sensor:
...
Currenlty I have to enter this in the yaml of the template trigger and this means I have to enter it to each template trigger which is a couple. This is hard to maintain and it comes with some for me unsolved issues:
Issue 1: I have to maintain the time intervals in the yaml
I appreciate that the template code itself (the linked sensor part of the template trigger) is coded in the yaml, so I can organize it in packages. But this does not apply to the trigger part of the template trigger: I’d like to use the automation UI to edit the time patterns and quickly turn my trigger on and off. I didn’t find a way to move the trigger part to an automation and refer to the template from there. If I try to update the derived entities from an automation, I get an error like “update function not implemented”.
Issue 2: Some trigger points have been missed
I observed situations where HA stopped to perform some of the time_patterns. I used a time slice of 2 seconds for MODBUS reads and I now extended it to 3 seconds.
Nevertheless, the mathematical outcome of this version implemented is as wished:
(graph will follow, I’m not yet allowed to post two attachments)
In this graph, the difference sensor is the blue line. At the falling edges you still can see small impulses in the result, but this is definitly an improvement and BTW it was perfect when modbus update times were <<1s. I don’t know where the delays now come from, maybe I will do some more modbus debugging.
The question I have here is:
Can I define a template (sensor or trigger) which is not updated based on the entities it is referring too, but updated by one or more automations?
BTW: What frequency I can expect HA to sample? I use 5 seconds for intensive measurements. I even succeeded with 1 second which was needed to analyse some strange sensor values. But for normal use a couple of minutes seems sufficiant. But what are the limits? For Modbus it seems less then 5 seconds (Allow unlimited scan_interval in modbus by janiversen · Pull Request #51471 · home-assistant/core · GitHub), but maybe for the rest 5 second is the minimum slice? I’m asking because my time slices currently are 3 seconds for modbus and 1 second for calculated updates.
Cheers!
Sumpfdotter