Multiscrape: Help needed to extract a value

Third party integrations
1

Hi,

A simple (but hard) one. My weather station has died and with it a pile of my automations. I need an alternate source for this data while I sort the problem/get a replacement (probably a good idea in any case to have a backup).

Ideally I’d scrape one or more public values as a data source using multiscrape but it looks like the data I want is in a java response or similar. I’m not too good at this so I would appreciate it someone with more experience could take 10 minutes to see if it is possible or not…

A sample URL is: Tempest

The value I’m after is BRIGHTNESS, expressed as “12345 lux” directly underneath the label.

I need this as with no weather station the brightness is currently 0 so it thinks it’s night and all our lights keep operating :wink:

Thanks!

2

OK…made some progress…I found a java query thing in the data that I can use to get a big hunk of data:

https://swd.weatherflow.com/swd/rest/better_forecast?callback=jQuery22409170868888429933_1695760092404&api_key=a8f5dbda-af0a-4b57-99b9-f10baa88f27b&build=76&lat=-41.17002&lon=174.83538&station_id=34978&units_temp=c&units_wind=kph&units_pressure=hpa&units_distance=km&units_precip=mm&units_other=metric&units_direction=kph&_=1695760092405

That brings back this mess:

jQuery22409170868888429933_1695760092404({"build_code":"005-s1-bt0.bwa0.bwg0.bwd0.bpop0.bst0-m0","client":{"base_url":"https://s3.amazonaws.com/tempest.cdn/assets/better-forecast/v9/"},"current_conditions":{"air_density":1.23,"air_temperature":7.0,"brightness":11054,"conditions":"Rain Likely","delta_t":1.0,"dew_point":6.0,"feels_like":6.0,"icon":"cc-rainy","includes_station_data":true,"is_precip_local_day_rain_check":false,"is_precip_local_yesterday_rain_check":true,"lightning_strike_count_last_1hr":0,"lightning_strike_count_last_3hr":0,"lightning_strike_last_distance":9,"lightning_strike_last_distance_msg":"7 - 11 km","lightning_strike_last_epoch":1692454791,"precip_accum_local_day":3.08,"precip_accum_local_yesterday":17.35,"precip_minutes_local_day":357,"precip_minutes_local_yesterday":957,"pressure_trend":"rising","relative_humidity":91,"sea_level_pressure":1006.1,"solar_radiation":92,"station_pressure":992.8,"time":1695760831,"uv":0,"wet_bulb_globe_temperature":7.0,"wet_bulb_temperature":6.0,"wind_avg":8.0,"wind_direction":195,"wind_direction_cardinal":"SSW","wind_direction_icon":"wind-rose-ssw","wind_gust":14.0},"forecast":{"daily":[{"air_temp_high":9.0,"air_temp_high_color":"0087d2","air_temp_low":6.0,"air_temp_low_color":"0053ee","conditions":"Rain Likely","day_num":27,"day_start_local":1695726000,"icon":"rainy","month_num":9,"precip":5.18,"precip_icon":"chance-rain","precip_probability":90,"precip_type":"rain","relative_humidity":88,"relative_sunshine_duration":5,"sunrise":1695751385,"sunset":1695795687,"wind_avg":21.0},{"air_temp_high":14.0,"air_temp_high_color":"4fe25b","air_temp_low":9.0,"air_temp_low_color":"0087d2","conditions":"Rain Likely","day_num":28,"day_start_local":1695812400,"icon":"rainy","month_num":9,"precip":2.08,"precip_icon":"chance-rain","precip_probability":80,"precip_type":"rain","relative_humidity":77,"relative_sunshine_duration":29,"sunrise":1695837682,"sunset":1695882148,"wind_avg":12.0},{"air_temp_high":14.0,"air_temp_high_color":"4fe25b","air_temp_low":8.0,"air_temp_low_color":"006ae4","conditions":"Rain Likely","day_num":29,"day_start_local":1695898800,"icon":"rainy","month_num":9,"precip":3.75,"precip_icon":"chance-rain","precip_probability":90,"precip_type":"rain","relative_humidity":76,"relative_sunshine_duration":42,"sunrise":1695923980,"sunset":1695968610,"wind_avg":27.0},{"air_temp_high":12.0,"air_temp_high_color":"14cb99","air_temp_low":8.0,"air_temp_low_color":"006ae4","conditions":"Thunderstorms Likely","day_num":30,"day_start_local":1695985200,"icon":"thunderstorm","month_num":9,"precip":9.7,"precip_icon":"chance-rain","precip_probability":90,"precip_type":"storm","relative_humidity":82,"relative_sunshine_duration":21,"sunrise":1696010278,"sunset":1696055072,"wind_avg":33.0},{"air_temp_high":14.0,"air_temp_high_color":"4fe25b","air_temp_low":7.0,"air_temp_low_color":"006ae4","conditions":"Rain Possible","day_num":1,"day_start_local":1696071600,"icon":"possibly-rainy-day","month_num":10,"precip":0,"precip_icon":"chance-rain","precip_probability":20,"precip_type":"rain","relative_humidity":80,"relative_sunshine_duration":67,"sunrise":1696096577,"sunset":1696141534,"wind_avg":24.0},{"air_temp_high":15.0,"air_temp_high_color":"4fe25b","air_temp_low":11.0,"air_temp_low_color":"00aabd","conditions":"Rain Likely","day_num":2,"day_start_local":1696158000,"icon":"rainy","month_num":10,"precip":7.27,"precip_icon":"chance-rain","precip_probability":80,"precip_type":"rain","relative_humidity":65,"relative_sunshine_duration":14,"sunrise":1696182875,"sunset":1696227997,"wind_avg":40.0},{"air_temp_high":12.0,"air_temp_high_color":"14cb99","air_temp_low":9.0,"air_temp_low_color":"0087d2","conditions":"Rain Likely","day_num":3,"day_start_local":1696244400,"icon":"rainy","month_num":10,"precip":1.54,"precip_icon":"chance-rain","precip_probability":60,"precip_type":"rain","relative_humidity":80,"relative_sunshine_duration":68,"sunrise":1696269174,"sunset":1696314459,"wind_avg":24.0},{"air_temp_high":14.0,"air_temp_high_color":"4fe25b","air_temp_low":9.0,"air_temp_low_color":"0087d2","conditions":"Partly Cloudy","day_num":4,"day_start_local":1696330800,"icon":"partly-cloudy-day","month_num":10,"precip":0,"precip_icon":"chance-rain","precip_probability":10,"precip_type":"rain","relative_humidity":75,"relative_sunshine_duration":65,"sunrise":1696355474,"sunset":1696400922,"wind_avg":25.0},{"air_temp_high":15.0,"air_temp_high_color":"4fe25b","air_temp_low":12.0,"air_temp_low_color":"14cb99","conditions":"Partly Cloudy","day_num":5,"day_start_local":1696417200,"icon":"partly-cloudy-day","month_num":10,"precip":0,"precip_icon":"chance-rain","precip_probability":10,"precip_type":"rain","relative_humidity":66,"relative_sunshine_duration":47,"sunrise":1696441773,"sunset":1696487386,"wind_avg":31.0},{"air_temp_high":16.0,"air_temp_high_color":"81d618","air_temp_low":12.0,"air_temp_low_color":"14cb99","conditions":"Rain Possible","day_num":6,"day_start_local":1696503600,"icon":"possibly-rainy-day","month_num":10,"precip":0,"precip_icon":"chance-rain","precip_probability":20,"precip_type":"rain","relative_humidity":65,"relative_sunshine_duration":50,"sunrise":1696528073,"sunset":1696573850,"wind_avg":29.0}],"hourly":

<SNIPPED - WAAAAAAAAAAAY TOO LONG>

Of that I can see the data I want:

"brightness":11054

How the heck do I pull that snippet out of that mess? Basically I know I need to find this bit:

"brightness":11054,"conditions"

Then replace [“brightness”:] and [,“conditions”] with [nothing] so that only ‘11054’ (or whatever the value is) is left…but…how?

3

OK…slow but steady…using:

value_template: '{{ value.split(",\"conditions\":")[0]}}'

I can get down to this now:

jQuery22409170868888429933_1695760092404({"build_code":"005-s1-bt0.bwa0.bwg0.bwd0.bpop0.bst0-m0","client":{"base_url":"https://s3.amazonaws.com/tempest.cdn/assets/better-forecast/v9/"},"current_conditions":{"air_density":1.23,"air_temperature":7.0,"brightness":14478

So the last stumbling block: how do I take the end of that, taking whatever is there until I hit the “:”? Whatever that is will be the value I need…

Sigh…so close and yet so far…

4

Went down a JSON rabbit hole…would have been an easy solution, but it’s not proper JSON…too much noise and too much curly braces everywhere…so back to trying to snip the end value off the string…

5

You could try using regex:

{{ value | regex_findall('\"brightness\":(\d+)') | first }}

However, the Weatherflow HACS addon might be a simpler solution.

6

Nice!

I was using the Weatherflow addon but my weather station seems to have died…so I need to scrape the data from another station to keep my automations going…and then once the rain stops I can get on the roof, retrieve the weather station and see if it is salvageable or needs to be returned…

:slight_smile: