Support for Environment Canada platforms

Third party integrations
104

UPDATE:

ENV CAN has confirmed that there are apparently periods of time where dd.weather.gc.ca is very slow in responding to queries…bandwith is too low…No data is returned before a timeout occurs and components cant be initialized. No Fail2Ban.

Env. Can. is presently trying to resolve this responsiveness issue. Some of us may occasionally experience this issue until throughput is enhanced by Environment Canada.

Env. Can strongly advises and encourages use of the AMQP (Advanced Message Queuing Protocol) . to access data on the dd.weather.gc.ca data.

Excahange message in French below.

c |### DPS-Client (EC)|7:47 AM (42 minutes ago)||
to me, DPS-Client

|

Bonjour Guy-Paul,

Je vous remercie de votre réponse rapide.

Il se peut en effet qu’il y ait des périodes de lenteur sur le Datamart du SMC, nous travaillons fort actuellement pour remédier à ces enjeux de bande passante.

Toutefois, j’aimerais profiter de l’occasion pour vous inciter fortement à modifier votre mode d’accès aux données vers le service AMQP (Advanced Message Queuing Protocol) .

Ce fil de données , mis en place par ECCC, permet non seulement à l’utilisateur d’être informé des produits disponibles au fur et à mesure qu’ils sont publiés, mais aussi de les recevoir automatiquement. Ce qui allège la pression sur les serveurs des 2 parties.

Le service peut filtrer des ensembles spécifiques de fichiers (alertes météo, observations, données de modèles, etc.), permettant des notifications uniquement pour les produits d’intérêt pour l’utilisateur.

C’est la méthode recommandée pour récupérer les données du Datamart du SMC .

La documentation de ce service se trouve sur la documentation MSC Datamart AMQP documentation.

Je demeure disponible pour vous aider dans cette migration ou pour répondre à vos questions.

Cordialement,

Sandrine

PS: je rappelle aussi qu’il est primordial de s’abonner à la liste de diffusion pour être averti de tout changement ou enjeu dans l’accès aux données du Datamart du SMC: https://lists.ec.gc.ca/cgi-bin/mailman/listinfo/dd_info

–

Sandrine Edouard

Coordonnatrice nationale, Accès aux données ouvertes

National coordinator, Open Data Access

Applications en traitement de données environnementales –

Environmental Data Processing Applications

Centre météorologique canadien - Canadian meteorological centre (CMC)

Environnement et changement Climatique Canada - Environment and Climate Change Canada

http://www.ec.gc.ca/

Courriel/email: [email protected]

Téléphone/Phone: 514-421-4729

2121 Trans-Canada N. Suite 230

Dorval, Québec

CANADA H9P 1J3

105

To facilitate the retrieval of timely data on the MSC Datamart, the Meteorological Service of Canada (MSC) has set up a data wire for announcing file availability on the MSC Datamart. This data wire uses the ‘Advanced Message Queuing Protocol’ (AMQP) protocol, making it possible not only to be notified of the availability of products on the MSC Datamart as they are published, but also to receive them automatically.

Timely updates without the need for frequent polling? Sounds good.

1 Like
106

I also noticed my HA doing a lot of DNS lookups of dd.weather.gc.ca. On my machine it was doing these lookups (followed by TCP connections on port 443) every 30 seconds. After adding scan_interval: 600 to the weather component the requests have slowed down to once every 10 mins, but I suspect most people won’t use scan_interval, letting HA try to update the component frequently.

@michaeldavie correct me if I’m wrong but it looks like the env_canada library seems to poll for updates every 30 seconds since those limits were moved out of the HA integration.

107

The env_canada library doesn’t do any polling on its own, it only reaches out when the __init__ or update functions are called. The update function has a limit of 2 calls per 60 seconds, which is perhaps what you were seeing.

The sensor component doesn’t currently have a refresh period of its own, so I believe that it’s using the default for all sensors of 1 minute. I’m open to suggestions as to a better default; so far I’m leaning towards @123’s suggestion of 10 minutes.

108

10 minutes sounds reasonable to me.

To be clear, the problem I experienced was with initializing values at a HA restart. All 3 Env Can platform components exceeded the expected time and then exceeded the hard limit . Reaching a hard limit results in an un-trapped error and is reported as such in the log file.

There are no subsequent retries and Env Can module values are shown as 'unavailable" until a restart with successful access.

Given that Env Can reports having bandwith issues and in such an instance, the MSC Datamart ( https://dd.weather.gc.ca. is unresponsive, failure to initialize values in any of the 3 components is quite possible. It would be great if the code was to trap initial errors and periodically retry to initialize a failing component.

109

@michaeldavie
I am receiving errors as follows. It seems the env_canada weather forecast sometimes contains more than the 255 character limit for Home Assistant states. I won’t get a forecast until they change it to something shorter. Can this be changed to parse the forecast and only write the first 255 characters to the entity_id.state, or some other way? Is this an issue for others, do I need to report it elsewhere? Thanks.
Error message:

homeassistant.exceptions.InvalidStateError: Invalid state encountered for entity id: sensor.forecast_2. State max length is 255 characters.
110

Thanks, I encountered that the other day with a template sensor I made as well. I’ll modify the integration to truncate the state of the sensors to 255 characters.

@balloob, what’s the correct way to implement scan_interval now? I’d like to increase the polling period of a component to 10 minutes, while leaving the third-party library intact.

111

Thanks for the attribution but I believe a 10-minute interval was davidbb’s suggestion (back in July).

As mentioned previously, I’ve been using a 30-minute interval for years with no adverse effects. The weather forecasts are typically updated every hour, or thereabouts, and weather warnings usually come in advance of the event (like over 30 minutes in advance). Can’t recall a time when the weather became severe without having already received a warning for it (using a half-hour polling interval).

112

Thanks for the quick reply and your work on this integration. I use it as a trigger to text myself and family when severe weather is possible, it works well, much appreciated.

113

I thought I should post this in case anyone has a similar problem. I am running Home Assistant 0.101.3 on a Raspberry Pi with Hassbian (although as it is now deprecated it is really just a Raspian Lite install in a virtual environment as I have updated manually).

When trying to add the Environment Canada integration I was getting an error message in my logs that indicated a failure to compile the Pillow library because of a missing dependency: jpeg.

I had run the commands in the documentation but after searching for the compilation error I found the suggestion to try:

sudo apt-get install libjpeg-dev zlib1g-dev

Upon re-staring Home Assistant the Environment Canada integration installed and is now working perfectly.

I hope this helps anyone else in the same situation.

114

It looks like the sensor component has been using a scan interval of 10 minutes for the last 4 months: https://github.com/home-assistant/home-assistant/blame/dev/homeassistant/components/environment_canada/sensor.py#L26

Maybe I just need to add this to the weather component?

115

Hey, if anyone is interested, it’s possible to get Yesterday’s snowfall from the weather page for your location if its available. Could be a good ADD for the Environment Canada Sensor as well. :slight_smile:

Using a scrape sensor (see below), replace "PP-NNN" with your page in https://weather.gc.ca/city/pages/PP-NNN_metric_e.html

ie. https://weather.gc.ca/city/pages/on-118_metric_e.html for Ottawa

Sensor:
  - platform: scrape
    resource: https://weather.gc.ca/city/pages/PP-NNN_metric_e.html
    name: Environment Canada yesterday's snowfall
    select: 'section:nth-child(6) > details > div > div:nth-child(3) > dl > dd'
    value_template: '{{ ((value.split(" ")[0]) | float ) }}'
    unit_of_measurement: "cm"
1 Like
116

Interesting, that data doesn’t seem to be in the XML file at all.

117

@michaeldavie:

Probably pulled from https://dd.weather.gc.ca/observations/xml/ON/yesterday/yesterday_on_20191115_e.xml

A more complex XML structure to parse…

118

Can I request a change to the State value of the Advisories, Warnings, Watches, Statements?

It would be super helpful to have a state value (perhaps None) when there are no Advisories, Warnings, Watches, or Statements.

Currently the empty state value does not allow the use of state_not in Lovelace.

I had a binary_sensor work around for this but that also appears to have broken at some point while updating HA. Perhaps as these sensors are no longer hidden or false when no alerts are issued.

Thanks

119

@rlongfield Good Idea,

Here’s a workaround that works with the current release of HA 101.3.

  bad_weather:
    value_template: >-
      {% set alerts = [ is_state('sensor.advisories', ""), 
                        is_state('sensor.statements', ""),
                        is_state('sensor.watches', ""),
                        is_state('sensor.warnings', "")] %}
      {{ alerts | select('==', false) | list | count > 0  }}
1 Like
120

@pg.sylvestre

Thanks for the work around. Glad to have my warning card back up and working!

1 Like
121

Unfortunately, I’m not able to change this. I tried previously, and the maintainers asked me to stick with None when there’s no value. If Lovelace is not able to handle this properly, you could try raising an issue on the main project.

122

That example gave me a sense of déjà vu! :slight_smile:

123

Wow, I really wish they would consolidate this data a bit more. The components already take quite a while to load due to all the fetching and parsing required, and the fact that they use different geo-bounding for each dataset. I’ll take a look at this one and see if there’s value in pulling it in as well.