Scrape a temperature to get a number

[CaptTom (Tom)]

Just starting to play with the scrape sensor. For a simple test, I tried to scrape the WeatherUnderground page for my location. I was able to get the right selection, but it returns a string containing the numeric temperature PLUS the “degree” symbol.

So when I display it along with other temperature sensors I have, it’s taken as a string instead of a number.

I tried using value_template, but apparently I’m not getting the syntax right. Is that the easiest way to go? What would the syntax look like?

[jchasey (John Chasey)]

Pipe the string into a float, e.g.:

value_template: '{{ value | float }}'

[CaptTom (Tom)]

Hmmm, that looks like it’s going in the right direction, thanks!

But now it’s returning 0.0.

[123 (Taras)]

It’s returning 0.0 because that’s the default value float reports when it can’t convert the supplied string.

You can’t use float to convert a string containing non-numeric text. In the example below, the F in the string can’t be converted by float so it reports its default value of 0.0

If you provide a sample of what you have, I may be able to create a template that extracts the numeric portion. For example, this will extract the numeric portion of 72 °F

[jchasey (John Chasey)]

Try:

{{ value.rstrip("F") | float }}

You can replace F with whichever trailing symbol(s) are used.

[CaptTom (Tom)]

@jchasey

Try:

{{ value.rstrip("F") | float }}

You can replace F with whichever trailing symbol(s) are used.

This did it, thanks!!

The character to be stripped was the “degree” symbol.

[123 (Taras)]

Out of curiosity, was there no way to make the scrape sensor extract the numeric value only and exclude the rest?

Ignore the brain fart …

[CaptTom (Tom)]

Still one issue. It looks like the {{ value.rstrip("F") | float }} is returning a string, not a float.

I suspect this because when I use the history_graph card to display it, I get a “bar” instead of a line graph like I do for other temperature values:

[finity]

Here you said:

@CaptTom

The character to be stripped was the “degree” symbol.

But in the post above you have:

@CaptTom

{{ value.rstrip("F") | float }}

Does the value still have the degree symbol in after you strip the “F”?

[123 (Taras)]

@CaptTom

It looks like the {{ value.rstrip("F") | float }} is returning a string, not a float.

It can’t return a string. By definition, it returns a floating point number.

If float is unable to perform the conversion it returns 0.0 (a numeric value).

FWIW, your ‘stepped’ graph looks like my thermostat’s history chart and an outdoor temperature sensor (minus the bar chart you have).

[CaptTom (Tom)]

@123

FWIW, your ‘stepped’ graph looks like my thermostat’s history chart and an outdoor temperature sensor (minus the bar chart you have).

Exactly. Those three values are defined the same way in the same history_graph card. I’m trying to figure out why one displays differently from the “stepped” graph I want. I click on the new one and get valid numeric values (now that the degree symbol is gone.) I even renamed the event and restarted HA in hopes that it wouldn’t retain any previous definition as a string. The new values are all now numeric, and still it displays as that odd bar graph.

I even created a new history_graph card in case the old one somehow remembered the configuration, but still got the bar.

[CaptTom (Tom)]

Update: I deleted the HA database, as well as added
unit_of_measurement: "°F"
to the scrape platform definition in the configuration.yaml file (which I renamed again to PWS_temp.)

That got me two separate line graphs on the card. We’re getting closer!

I then changed the unit_of_measurement on the other thermometers to “°F” instead of “degrees.”

Now all three lines show up on the same chart! The trick was to make sure they were all using the same unit of measurement. Of course, I’ve only got a few minutes’ worth of data so far, so the chart isn’t that helpful yet.

[bluenote73 (bluenote)]

Hey, I’m trying to pull temperature from the WU station nearest me, but I don’t understand how to decode the format to properly reference the data. Could you actually post your config so maybe I can extrapolate from what you did? Sorry for being such an amateur

[CaptTom (Tom)]

@bluenote73

Could you actually post your config so maybe I can extrapolate from what you did? Sorry for being such an amateur

I ended up using the Weather Underground API via the Rest platform:

  - platform: rest
    name: pws_report
    json_attributes:
      - observations
    value_template: '{{ value_json["observations"][0]["obsTimeLocal"].title() }}'
    resource: https://api.weather.com/v2/pws/observations/current?stationId=my_station_id&format=json&units=e&apiKey=my_PWS_key
    scan_interval: 300

Then to get the attributes I used templating:

      pws_precip_total: 
        value_template: '{{ states.sensor.pws_report.attributes.observations[0].imperial.precipTotal }}'
      pws_precip_rate: 
        value_template: '{{ states.sensor.pws_report.attributes.observations[0].imperial.precipRate }}'
      pws_temp: 
        value_template: '{{ states.sensor.pws_report.attributes.observations[0].imperial.temp }}'
        friendly_name: "PWS Temp"
        unit_of_measurement: "°F"

As I recall, I was able to just scrape the page for a while, but WU changed the way the page coded the temperature and it stopped working. Basically, you look at the page, find the temperature, then scan through all the web page source for that number. It wasn’t obvious and I think the change made it almost impossible to extract it reliably.

The problem is, to use the API you need a key, which you can only get if you have a PWS.

[bluenote73 (bluenote)]

Thank you, kind of you to do all that. You’re right though, I don’t have an API key.
I actually (kind of) got this working, but the page seems to refresh after you go to it, and that updates the value. I guess HA doesn’t understand that, so it always pulls the wrong number that is populated to begin with (20.1). It’s too bad because the local PWS weather station is very accurate for me. Thanks anyways!

[bluenote73 (bluenote)]

Oh, actually, I realized it’s just giving me a value in F instead of C because they do some silly after the fact refresh. Hopefully I can somehow put a formula into the sensor so it just reports C.

[CaptTom (Tom)]

@bluenote73

I actually (kind of) got this working, but the page seems to refresh after you go to it, and that updates the value.

Right, it’s a pretty dynamic page. Hard to figure out what to scrape. It might be possible to locate the actual source of the data; maybe it’s in some file that you could pull down without grabbing the whole page.

I’d be curious to see how you’ve got it configured to grab the temperature. I recall struggling with temperature conversion too. It seems there are multiple places this can occur.

[bluenote73 (bluenote)]

In the end it actually ended up being quite simple, this works perfectly delivering a measurement in C because that’s what HA shows me. (Keep in mind, I’m scraping a particular PWS that is close to me, so if you had a different page in mind it might not be the same)

[CaptTom (Tom)]

Thanks, yes, that’s what I see, too. There are a bunch of values with a class named “wu-value-to.” It looks like the first one is the temperature. But on the page I’m looking at, there are 15 other values that use the same class. I guess the scraper just grabs the first one. Nice, elegant solution!

[alnavasa (A)]

having the same issue,
flowed the steps and still happening

  # zzzzz / health levels
  - platform: scrape
    resource:zzzzzzz
    select: "zzzzz div > section:nth-child(2) > div > div:nth-child(1) > div > div:nth-child(2) > div > div > span.AirQualityText--measurement--2AGlG.AirQuality--pollutantMeasurement--dlvtB"
    name: zzzz zzzz
    value_template: '{{ value.rstrip(" μg/m3") | float }}'
    scan_interval: 320