Im very aware that my use-case more have the character of a FR, as i need the already through the remote API requested weather-data survive a reboot (regardless of why a reboot is needed/done )
Many people live outside Main-areas, on countrysides ( which in many countries is not the first priority, after a heavy storm, massive rainfall or snow etc , so loss of internet-connection occurs several times a year just do to the weather , tree falls, "distribution-points ( basically providers switch boxes around the country gets flooded, damaged etc.) … so sometimes half a day to several days, i can not rely upon Cached(only)-cloud data, it’s as simple i can say it.
PS, if the electricity ( most often ) also many times is affected, it’s always the first that comes up again, but obviously a private backup , like solar-power, or diesel-generator is a comfy thing to have 
… And no , i live in a mobile-signal “shadow” , im lucky if i can get 3G on my phones, at home(not that i think it would help when such scenarios occurs ))
I’ve been trying to follow this discussion and I may still need some clarification…
Here is an example weather entity that returns the current hourly conditions AND in addition provides 24 hours of forecast conditions as attributes:
in the situation we are discussing am I correct that all of the “forecast:” attributes would be completely removed and we would need to create template sensors and call a service to convert those (once upon a time attributes) into individual hourly sensors in their own right whose state will contain the exact same info that was previously contained in the hourly attributes?
So each sensor would have a state of (using the first “forecast:” attribute from the above screen shot as an example)?:
I know that we were admonished to not talk about the “why?” but I’m just really having a hard time seeing the benefit. Especially if the integration API polling doesn’t change and we just need to pull the already polled and existing data from a cache somewhere to actually be able to use it?
I think it’s already covered in the thread, but the whys are basically:
- Avoid fetching data from remote API unless needed
- Avoid storing and moving around large-ish amounts of weather forecast data in the state machine
For the most case the forecast data is refreshed around the clock and then just thrown way, that is, if frontend is not displaying the weather forecast and no automation, template sensor or similar accesses the forecast.
The way the caching works is that forecast data is fetched from the remote API once within a time interval defined by each weather integration when the service weather.get_forecast is called.
I thought I read that the remote API is still called at the same interval as defined within the integration. So the number of API calls wouldn’t change?
The only difference being discussed is that the API data is cached somewhere and the service call would retrieve the already cached data and supply it to the state machine at that point.
But isn’t that exactly how every other entity/integration behaves as well?
I have many entities that get regular updates of data to the entities (as often as every few seconds depending on the integration/device) that I usually do nothing with so it gets “just thrown away”. But then occasionally I need to act on that data so it’s nice to have it in the state machine to be able use them in automations if needed.
I thought that was the whole purpose of the data. to have it in the state machine so as to monitor it and use if needed.
and I actually use the attribute data to be displayed in the front end in my weather cards. I guess it means to continue to use the forecast data in the weather cards I will need to create a sensor for every daily forecast that used to be in one place in the attributes of the weather entity?
but TBH, I’m not sure that the weather cards are set up to use the data from sensors. I believe the ones that I use require the data for the forecast to be contained in attributes.
Is there some huge hit in performance that has come to light that is caused by the data being moved around that is bad enough that we need to eliminate the data availability?
But also as mentioned above it seems that in order to get back the data we already have now we will need to create additional sensors that will by definition add that same data back into the state machine. But it will then have a larger impact on the DB since most users won’t set up the recorder to exclude the new sensors so all of the data that used to be in attributes (and to my understanding wasn’t included in the db) will now be written there instead.
Or am I missing something fundamental and not seeing the issues that are being caused now? I don’t think I have seen any hit in performance of my system and I regularly collect both hourly and daily forecast data from at least 3 weather integrations.
Yeah this seems completely pointless to me and is only making things more difficult for the end user.
Every time you open the web interface or the mobile app wakes, HA has to send the entire contents of the state machine to the browser or mobile to get things back up in sync. It’s impossible to avoid this because once the connection is lost, there is no way to know the system’s state without doing a full sync on reconnect. The more that’s in the state machine, the longer you will have to wait before you can interact with HA. There are knock-on effects that affect add-ons and many other places as well. Larger attributes use more disk I/O and memory, push API calls to the limits on some systems, and have recently triggered some failures/overload scenarios due to hitting maximum payload sizes.
This problem has been getting worse over time as more integrations and entities become available and are added to the system. Only so much can be done to optimize it (we have nearly run out of things we can do). Entities with significant attributes can take up as much space as 100 entities, so reducing them is essential to keeping this problem from deepening.
While throwing hardware at the problem is one solution, HA primarily targets RPi systems, so it’s essential to ensure that the HA scales well on these systems.
This makes sense and I have definitely noticed it, especially on slow mobile data connections. Definitely seems like a worthwhile change IMO
What happens when I open my main dashboard with a forecast card on it?
The data for this card has to be sent anyway does it not?
How would it update otherwise?
Could it be that the syncing of the state machine’s state is blocking (during the loading of the app, as mentioned, meaning nothing will happen until that’s done), whereas this would be non-blocking (i.e. load async because it will be a separate service call after the fact)?
Obviously I’m taking an educated guess, so keen to hear what the core devs say.
This makes sense, on the other hand, my definition of a HomeAutomation system is that atleast the essential components of such system should somehow be easy accessible, stored etc. , but following this Community i’ve notice that many people “overload” their HomeAutomation system with integrations , that actually (in my opinion) belongs to an i.e “Media.system/archiving system etc. etc”
As mentioned earlier( atleast the way i use/see the weather-forecast integration) , it’s the basic info,imited in range/details, (Not as much as i , my self, can retrieve with a manual API call to i.e met.no,) which i need for i.e automations/template to control my Home , i.e Heating/Cooling system, Power.consumptions/savings, irrigation and other essential integration/functions in a Home environment.
A HomeAutomations system should prioritize these, such functions.
The limited data of the i.e 24hours/7days arrays(which is “recycled”) don’t ever grow beyond whats visible in the “Attributes” atleast should’t , how ever im sure there are lot’s of use-cases with the ever growing added/supported integrations, and what some people find it cool/funny etc to Add to their homeautomations system ( these people need to understand the “Cost” of their use of and dimension of their behavior/wishes.
Sacrificing an highly essential Integration ( weather-integration ) with limited( and in most cases fixed data size ) Very useful information in i.e a Homeautomation system, for the sake of god knwos what people stuff into i.e HA, just because they can, integrations, data etc. which in my opinion doesn’t belong to a HomeAtomations system.
I don’t know how it’s “penetrated/concluded” that the weather-integration is “a burden”, for the state-machine( I sincerely doubt) , but that the new proposed/decided re-build of the integration, will ofcause leave “space/resources” for other non.essential" integrations etc.
Key word is prioritization, where each one have to make own decisions, and knowing the consequences, SURE mobile connection/use cases, is somehow very common, but hardly the ground/foundation a HomeAutomation core should be build upon , … if some one have problems with wifi interferrence, slow phone, chocked browser etc, it has absolutely nothing to do with the HomeAutomation system, AND/OR the very litlle data from i.e stored weather-forecast data.
People have to know this, and prioritize their system cording to their needs.IF they want their whole “lib” of movies/music or tons of “state-details” or tons of colored-dimible-lights etc. etc. Everything comes with a Costs
It seems to me that the focus, in my opinion, leaning towards a “Play Gound” NOT an Prioritized HomeAutomation system.
Sacrificing a “simple” function/feature ( highly usable in controlling essential “States” in a Home ) , for the sake of i.e Mobile browser use-cases, or other non-essential use-cases/integrations in a HomeAutomation system, shold not be the way forward for HA, in my opinion.
HA will never ever be able to fulfill a Total, all-round use.case, network-communication, hardware, common interface knowledge limitations etc etc. will always.be a factor, so “some” people might considered the need for i.e an “Additional” HA installation, if some have issues which HA ( or in particular, their network environment, mobile, browser) can’t “cope” with
Already when it was announced that “Attributes” would be removed ( unconditionally ), some 3rd part-integrations-devs , started to “move” Attributes into individual sensors/entities , so i’ve been wondering, how many additional entities do we have already now, do to this announced decision.
States are the essential condition in a HomeAutomation system ( even it is Forecast-data )
If i should choose between flood/Lightning-announcements VS upcoming-song-in-a-playlist ( i know, Silly example ) However, my purpose using HA is for controlling my Home
bdraco , im sure you have made an overview of the Core-Components in HA and their impact on the DB, but obviously i can’t know which is HA’s prioritizes
So tha’t why i constantly gets “notifications” that ( Ha is “updating sensors” ) ? , even thou i don’t have the HA-App opened ( seems more like to me this is a “background function” regardless if i haven’t open the APP for hours), … if all this boils down to a “Mobile” issue, then it is what it is , NOT an HomeAutomation issues caused by “Weather-Forecast-Attributes”
PS: Essential to mention is maybe that i only use my phone on “Local Connection”, and very seldom few time a day , either just to verify certain “states” i don’t have a utopia-illusion, that i would ever be able to load ALL “States” from ALL integrations in an instance if im abroad, or just “away” from home.
But i would feel kind or more “safe” if an weather-forecast-integration fetch the data from the API, and could keep function until all the data it just received from the API, is “staled” ( cording to the “TimeStamp” in each forecast-entriy , atleast it would buy me/the system some time if problems occur retrieving new data from the API ( even thou im aware that forecasts-change by the hour/day, so it might not be the “latest” prognoses, BUT the system will keep running, for 23-hours ahead, and 6-10 days, what ever is retrieved. NOT stop function because it “missed” a few API calls
This really has nothing to do with the topic at hand
Well, I’m going to slowly start pecking away at this total rewrite if anyone wants to help.
I know, It just have to do, how i consider a weather-forecast-integration should function/have priority in a HomeAutomation System
( fetch the data from remote-API, store it in DB/File, Keep function until all forecast-entries is due , just in case … ) it’s better it keeps working, than stops working bc it couldn’t retrieve new/updated info from the remote-API , i would then get a “warning” (can’t retrieve new data) and have time to investigate, and take precautions, … it’s not like weather-forecasts is so damn unreliable that the forecast (prognoses) changes rapidly and extreme, they tend to be quite “stable” in near future.
Why would you need all this information in the state machine?
What is your use case for this forecast data?
Its going to load the forecast data async and (re-)render the card when its available
Eventually it does but not until its needed to render the specific card
If it was already in the state machine it would be loaded on the initial connection resync. Which is effectively blocking because almost everything needs the data to render
The answer is a bit mirky but since many cards/panels needs data from state machine (or the registries, or config, or some other place that could have changed since the connection was lost) its effectively “blocking” (even if it technically isn’t for some places) until the state is restored from the backend after reconnect.
There are ways to speed up getting back in sync with the backend, but they require the backend to keep the state of the state machine at a specific time and be able to replay only the changes between when the connection cut off and when it was restored. That might be faster, but it’s a significant workload (memory and cleanup of old data), adds considerable complexity, and has bug risks for the backend, so it’s probably a non-starter without increasing hardware requirements.
Am I missing something or is the weather forecast the most tiny and compressible part of the frontend?
Also crazy idea but what if states could be streamed in on the initial load instead of blocking
( 1 Simplified use-case)
IF Tomorrow the outside temperature will Dropp to -15 degree @ 00.05am
( i.e from current +10 degree) and during this time, from “NOW” obvoius the actually(current) temperture might/will differ from “Prognoses”, this will also be ajusted in template( by local devices) !
And temperature in “heated-water-tanks is == x”
( or , if inside temperature is == x)
… or both above
start system-heating the house @ 10pm today
( there is Always some delays in i.e warming up a house(a totally individual factor), also depending upon, whether it’s water-heated(what’s the water storage temperature currecntly etc. etc.) or direct electricity heated, how many watts do you have available, to reaching/securing to decired temperature etc. with in the actual timespan ) Even here is ofcause current temperatures(local devices) involved in the equation.
If you only use Current-data ( Local-devices ) for your i.e heating-system, the “house-hold” of your resources will be ineffective, you burn alot resources, maybe even at wrong time, because "suddently, out of nowhere, the i.e temperature dropps rapidly, below the amount you have available, or the capasity to somehow instantly get the temperature to desired level ) …
-
Sunny/clouded
-
Lightning/Storm
-
Heavy precipitations
-
Temperatures
-
Sunny/Clouded is even useful if one have solar systems
-
Coming Price of electricity , kind of “forecast” that eventually becomes history in the state-machine ( Removed from Attributes soon)
etc. etc.
ALL Very " Important " factors in a HomeAutomation system(Server) , and im not Talking about what and how fast my Mobile can retrieve the DATA, because THAT is absolutely a non-essential factor for how a HomeAutomation System(Local-Server) Should work
Im not going into a debate, of all the issues people have with their wifi, or external UI access to their HA, because thats absurd, considered that 99.9% of HA users ( including me ) hardly have any clue what “LEVEL” my Router currently handles the wifi traffic, WIFI’g-Level quite often ? (do to limitation in 1 or few devices on the network ? ) , AND remote access !!!, what over metered-mobile-connection ?, on a sunken free wifi-network in outer town, or at friend/airport wifi ? … forget it, and often not mentioned in many use-cases here in the Forum somehow. AND hopefully nothing which have priority, or taken into consideration when buildning the Core
PS: I know your question was to finity 
but with the new action section in trigger based templates you can create a binary sensor for that temperature drop. You don’t need to have all the data available in entity states or attributes.
Based on the same trigger, you can also create (and update) a sensor which gives the precipitation of the next x hours, or a binary sensor which is on of there is precipitation forecasted etc.
So therefor I wanted to know why there is this need to actually create template sensors for each hour of the forecast.
Yes i know, and wait with further “test/templates” until final solution is in place, but as mentioned elsware, we are just replacing attribute with a bunch of sensors/entities , but, howvever if the sensors can just survive a reboot(with the individual states intact, and the template/automatioin working regardless of loss of internet-connection, and my system not hangs during startup, for same reason( no internet ) im content, my system will be able to handle lots more entities/sensors, so just have to “ajust” for the lack of Attributes, in the reference … And hope the system comes up and running, spite the lack of, and unable to preform remote API calls ( i know a few integrations that does “hang” the start-up 
Wait, i somehow missed this point, does that mean i can create a( or 3-5) binary sensors “on the fly” ( ofcause i then need an automation that goes through the “latest” forecast(from the latest remote-API call), and “pick relevant” trigger points, to create these sensors, an then update these sensors(during each new remote API call ), so when/if my system goes down(or i manually reboot), and eventually comes up again, with no internet( I DO expect these sensors have the same State, after a reboot) , so it’s still “on” ? ( and ofcause updated if im lucky to/or when i get a remote-API cal through)
My god we will see lots of Topics in this Forum in the future, tweaking these automations/tempates/binaries-sensors, which aims to control the automations/templates for the actual Fucntion we want to address