Scrape sensor for more entities

michal.bohac · January 6, 2023, 10:00pm

Hi guys,
I need advice on how to find out several different pieces of information from a website. I tried via multiscrape.yaml but no luck.

I need to “scan” EUR, GBP and USD rates. Everything for the current day from CNB (Czech National Bank).

I tried it in configuration.yaml like this, but it just keeps showing me only the first entity EUR rate. I don’t see those other entities anywhere.
I look in “developer tools” → “states” → and here I filter the entities.
And I still only see the first one.
Anyone has some idea?

scrape:
  - resource: https://www.cnb.cz/cs/financni-trhy/devizovy-trh/kurzy-devizoveho-trhu/kurzy-devizoveho-trhu/
    sensor:
      - name: Kurz "CZK/EUR"
        select: "td"
        index: 29
        value_template: '{{ value | replace (",", ".") | float }}'
        unit_of_measurement: "CZK/EUR"
        #scan_interval: '08:00:00'
   
      - name: "Kurz CZK/USD"
        select: "td"
        index: 153
        value_template: '{{ value | replace (",", ".") | float }}'
        unit_of_measurement: "CZK/USD"
        #scan_interval: '08:00:00'

      - name: "Kurz CZK/GBP"
        select: "td"
        index: 158
        value_template: '{{ value | replace (",", ".") | float }}'
        unit_of_measurement: "CZK/GBP"
        #scan_interval: '08:00:00'

By the way, I tried multiscrape.yaml I had it like this and it did the exact same thing, I only saw the first entity:

resource: https://www.cnb.cz/cs/financni-trhy/devizovy-trh/kurzy-devizoveho-trhu/kurzy-devizoveho-trhu/
scan_interval: 3600
sensor:
    - unique_id: kurz_cnb_eur
      name: Kurz ČNB EUR
      select: "tbody > tr:nth-child(6) > td:nth-child(5)"
      unit_of_measurement: "EUR"
resource: https://www.cnb.cz/cs/financni-trhy/devizovy-trh/kurzy-devizoveho-trhu/kurzy-devizoveho-trhu/
scan_interval: 3600      
sensor:
    - unique_id: kurz_cnb_gbp
      name: Kurz ČNB GBP
      select: "tbody > tr:nth-child(32) > td:nth-child(5)"
      unit_of_measurement: "GBP"
resource: https://www.cnb.cz/cs/financni-trhy/devizovy-trh/kurzy-devizoveho-trhu/kurzy-devizoveho-trhu/
scan_interval: 3600   
sensor:
    - unique_id: kurz_cnb_usd
      name: Kurz ČNB USD
      select: "tbody > tr:nth-child(31) > td:nth-child(5)"
      unit_of_measurement: "USD"

Thanks a lot if someone can help…

danieldotnl · January 10, 2023, 1:03pm

Take another look at the multiscrape readme. Your config is not correct, you should not repeat the resource and scan_interval etc.

Troon · January 10, 2023, 1:53pm

I think the quote is causing your problem. Should be before Kurz.

      - name: Kurz "CZK/EUR"
--------------^----^

Here’s how you really want to do it, though, using the XML file they provide. Much better than scraping for an arbitrary <td> that will break if they add / remove a currency or change their layout.

rest:
  - resource: https://www.cnb.cz/cs/financni-trhy/devizovy-trh/kurzy-devizoveho-trhu/kurzy-devizoveho-trhu/denni_kurz.xml
    scan_interval: 28800
    sensor:
      - name: Kurz CZK EUR
        unit_of_measurement: 'CZK/EUR'
        device_class: monetary
        value_template: "{{ value_json.kurzy.tabulka.radek|selectattr('@kod','eq','EUR')|map(attribute='@kurz')|first|replace(',','.') }}"

      - name: Kurz CZK GBP
        unit_of_measurement: 'CZK/GBP'
        device_class: monetary
        value_template: "{{ value_json.kurzy.tabulka.radek|selectattr('@kod','eq','GBP')|map(attribute='@kurz')|first|replace(',','.') }}"

      - name: Kurz CZK USD
        unit_of_measurement: 'CZK/USD'
        device_class: monetary
        value_template: "{{ value_json.kurzy.tabulka.radek|selectattr('@kod','eq','USD')|map(attribute='@kurz')|first|replace(',','.') }}"

michal.bohac · January 11, 2023, 1:46pm

Thanks very much Troon for your help, now everything is working as I expected - and even better with the XML profile!

Really appreciate your time…have a nice day

blavak68 · January 12, 2023, 12:55pm

Hi can you help me pls, with this xml

<gesmes:Envelope>
<gesmes:subject>Reference rates</gesmes:subject>
<gesmes:Sender>
<gesmes:name>European Central Bank</gesmes:name>
</gesmes:Sender>
<Cube>
<Cube time="2023-01-11">
<Cube currency="USD" rate="1.0747"/>
<Cube currency="JPY" rate="142.61"/>
<Cube currency="BGN" rate="1.9558"/>
<Cube currency="CZK" rate="24.027"/>
<Cube currency="DKK" rate="7.4375"/>
<Cube currency="GBP" rate="0.88673"/>
<Cube currency="HUF" rate="399.1"/>
<Cube currency="PLN" rate="4.6819"/>
<Cube currency="RON" rate="4.9335"/>
<Cube currency="SEK" rate="11.2783"/>
<Cube currency="CHF" rate="0.9967"/>
<Cube currency="ISK" rate="154.3"/>
<Cube currency="NOK" rate="10.738"/>
<Cube currency="TRY" rate="20.1793"/>
<Cube currency="AUD" rate="1.5588"/>
<Cube currency="BRL" rate="5.584"/>
<Cube currency="CAD" rate="1.4429"/>
<Cube currency="CNY" rate="7.2807"/>
<Cube currency="HKD" rate="8.3974"/>
<Cube currency="IDR" rate="16612.34"/>
<Cube currency="ILS" rate="3.7104"/>
<Cube currency="INR" rate="87.812"/>
<Cube currency="KRW" rate="1341.01"/>
<Cube currency="MXN" rate="20.4957"/>
<Cube currency="MYR" rate="4.6975"/>
<Cube currency="NZD" rate="1.6912"/>
<Cube currency="PHP" rate="59.013"/>
<Cube currency="SGD" rate="1.4316"/>
<Cube currency="THB" rate="35.895"/>
<Cube currency="ZAR" rate="18.2827"/>
</Cube>
</Cube>
</gesmes:Envelope>

i habe problem with gesmes, i try it:

rest:
  - resource: https://nbs.sk/export/sk/exchange-rate/latest/xml
    scan_interval: 28800
    sensor:
      - name: Kurz USD (nbs)
        unit_of_measurement: 'USD/EUR'
        device_class: monetary
        value_template: "{{ value_json.gesmes.Cube.Cube.Cube|selectattr('@currency','eq','USD')|map(attribute='@rate')|first|replace(',','.') }}"

after HA reload is in logs this error

Logger: homeassistant.helpers.template
Source: helpers/template.py:589
First occurred: 13:54:31 (2 occurrences)
Last logged: 13:54:31

Template variable error: 'dict object' has no attribute 'gesmes' when rendering '{{ value_json.gesmes.Cube.Cube.Cube|selectattr('@currency','eq','USD')|map(attribute='@rate')|first|replace(',','.') }}'
Template variable error: 'dict object' has no attribute 'gesmes' when rendering '{{ value_json.gesmes.Cube.Cube.Cube|selectattr('@currency','eq','GPB')|map(attribute='@rate')|first|replace(',','.') }}'

thx in advance

Troon · January 12, 2023, 2:48pm

You were close:

        value_template: "{{ value_json['gesmes:Envelope'].Cube.Cube.Cube|selectattr('@currency','eq','USD')|map(attribute='@rate')|first }}"

You don’t need the replace — we only had that in last time because the data was using “European” decimal commas.

blavak68 · January 12, 2023, 3:09pm

… replace I knew, but you helped me with ['gesmes:Envelope']

once more thx

michal.bohac · January 15, 2023, 7:50pm

Hi Troon again,
can you please be so kind and let me know, if you are able to read the currency values from this site?
It is a CSOB bank from CZ and I assume they are not using XML as the CNB CZ bank.

https://www.csob.cz/portal/people/exchange-rates

The CNB exchange rates are fine, but they are always lower than the CSOB bank exchange rates.
And as I use the CSOB bank, I would need to know more precisely their rates.

Thanks very much!

Troon · January 23, 2023, 3:58pm

I didn’t see this earlier because you replied to yourself not me and didn’t tag me, so no notification.

Took me 5 seconds to see this:

michal.bohac · January 25, 2023, 11:00am

Hello, well I am stupid, did not see that - sorry.
Now I am struggling to change the current sensor:

sensor:
      - name: Kurz CZK/EUR
        unit_of_measurement: 'CZK/EUR'
        device_class: monetary
        value_template: "{{ value_json.kurzy.tabulka.radek|selectattr('@kod','eq','EUR')|map(attribute='@kurz')|first|replace(',','.') }}"

I´m sure I have to change the last line, but don´t exactly know, what to put there…

sensor:
      - name: Kurz CSOB CZK/EUR
        unit_of_measurement: 'CZK/EUR'
        device_class: monetary
        value_template: "{{ value_json.ExchangeRate Date.Country ID|selectattr('@kod','eq','EUR')|map(attribute='@kurz')|first|replace(',','.') }}"

Amy idea please?

Troon · January 25, 2023, 11:17am

Here’s what you need. Looks like the XML URL changes daily, so I’ve included the date in the resource_template — hopefully that’ll work, depending on when they publish the new one each day:

rest:
  - resource_template: https://www.csob.cz/portal/lide/kurzovni-listek-old/-/date/{{ now().isoformat()[:10] }}/kurzovni-listek.xml
    scan_interval: 28800
    sensor:
      - name: Kurz CZK EUR
        unit_of_measurement: 'CZK/EUR'
        device_class: monetary
        value_template: "{{ (value_json.ExchangeRate.Country|selectattr('@ID','eq','EUR')|first)['FXcashless']['@Buy'] }}"

I’ve picked the Buy rate (Nákup) but you can change '@Buy' to '@Sale' or '@Middle' depending on which you need. Here’s how that search template is working:

michal.bohac · January 25, 2023, 12:12pm

Wow, that is absolutely perfect. Everything is working fine and let´s see if it works on the other days…
Thanks man very much, any way how to donate to you for your help :-)?

Troon · January 25, 2023, 1:03pm

If that’s solved your problem, please tick the Solution box on my post.

Strictly, you should tick my first post in the thread, as that answered the original question.