Multiscrape / Scrape from Tradingview

[Salve]

Hello Guys,
New try on Scraping
Can you help me on solving my issue

Thanks Daniel

I’m trying to scrape this number from Technical Analysis of Euro / U.S. Dollar (FX:EURUSD) — TradingView

My Code is:

  - resource: https://www.tradingview.com/symbols/EURUSD/technicals/
    scan_interval: 200
    log_response: true
    sensor:
      - unique_id: eurusd
        name: cci
        select: "div:nth-child(1) > div.tableWrapper-Xoorn8RH > table > tbody > tr:nth-child(4) > td:nth-child(2)"
        on_error:
          log: warning
          value: last

And I’m getting this Logs:

2023-01-22 20:34:48.425 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # New run: start (re)loading data from resource
2023-01-22 20:34:48.425 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Deleting logging files from previous run
2023-01-22 20:34:48.432 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Rendered resource template into: https://www.tradingview.com/symbols/EURUSD/technicals/
2023-01-22 20:34:48.433 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Request data from https://www.tradingview.com/symbols/EURUSD/technicals/
2023-01-22 20:34:48.433 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Executing page-request with a get to url: https://www.tradingview.com/symbols/EURUSD/technicals/.
2023-01-22 20:34:48.442 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # request_headers written to file: page_request_headers.txt
2023-01-22 20:34:48.449 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # request_body written to file: page_request_body.txt
2023-01-22 20:34:48.724 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # Response status code received: 200
2023-01-22 20:34:48.727 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # response_headers written to file: page_response_headers.txt
2023-01-22 20:34:48.732 DEBUG (MainThread) [custom_components.multiscrape.http] Scraper_noname_0 # response_body written to file: page_response_body.txt
2023-01-22 20:34:48.733 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # Loading the content in BeautifulSoup.
2023-01-22 20:34:49.021 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # page_soup written to file: page_soup.txt
2023-01-22 20:34:49.021 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Scraper_noname_0 # Data succesfully refreshed. Sensors will now start scraping to update.
2023-01-22 20:34:49.021 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 0.596 seconds (success: True)
2023-01-22 20:34:49.021 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # cci # Start scraping to update sensor
2023-01-22 20:34:49.036 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_0 # cci # Tag selected: None
2023-01-22 20:34:49.037 WARNING (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # cci # Unable to scrape data: Could not find a tag for given selector 
Consider using debug logging and log_response for further investigation.
2023-01-22 20:34:49.038 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_0 # cci # On-error, keep old value: None
2023-01-22 20:34:49.039 DEBUG (MainThread) [custom_components.multiscrape.entity] Scraper_noname_0 # cci # Sensor updated and state written to HA

[Salve]

Hey Guys,
tried quite alot but still without success
Any Ideas?
Best regards

[danieldotnl (Daniel)]

Seems like the value is dynamically loaded by javascript that’s executed in the browser. Multiscrape can’t do that. Dive into the developer tools and try to find out which response (probably json) contains the value you are looking for.

[Salve]

Hello Daniel,
ok thanks for the analysis.
Unfortunately i failed finding an Json response. Doing something wrong. Can you support once more?
Best regards

[Salve]

Hello Daniel
i found a alternative site on tradingview:

Major Currency Pairs — Rates and Performance — TradingView

I managed to scape the Header “Ask”
Still trying to scrape the CCI Value

Could you give this another try? please

Best regards

[Troon]

I think you’re out of luck on this one — it seems to pull its rates via a websocket connection:

Same technique the HA UI uses, ironically, but scrape can’t help you here.

[Salve]

Hello Troon,
Any idea/alternative how I could fetch this data?

My next try is to copy the data to an csv (or json) local file an import to home assistant.
But that’s a lot of work (nothing Assistant like)

Best regards

[Salve]

Hey Daniel,
new try
I found another homepage:

EUR USD Technical Analysis - Investing.com

Multiscrape works great. Only i would like to scrape Daily not “standard setting” hourly .
Can i use “form_submit” to change to daily

Alternativ on: Forex Technical Indicators - Investing.com
there is a timeframe dropdown

Would be great for a little support
Thanks

  - resource: https://www.investing.com/technical/indicators
    scan_interval: 200
    form_submit:
      submit_once: false
      resource: https://www.investing.com/technical/indicators
      select: "Daily"
    sensor:
      - unique_id: investing
        select: "#curr_table > tbody > tr:nth-child(1) > td.right"
        name: investing

is not working

[danieldotnl (Daniel)]

Bad luck. If you choose “daily”, another request is executed in the background to this url: https://www.investing.com/instruments/Service/GetTechincalData
If you go to this url directly, you are just forwarded to the mainpage. I guess it checks if the cookie is sent in the request. You might try to add it in a header field, but it will probably expire.
Form-submit will not work since that button is just triggering javascript and not part of a form.

[Bonzenpaule]

@Salve any success in your quest? I’m trying to get the daily 200 SMA data through multiscrape, scrape or rest API. So far I couldn’t get anything to work.