Scraping actual price Dutch TTF Gas

Configuration
1

I tried to scrape the actual gasprice form the Dutch TTF Gasprices but get stuck with it.

I would appreciate if someone can give me a solution to solve my problem.

My YAML looks like this

  - platform: scrape
    resource: https://www.theice.com/products/27996665/Dutch-TTF-Gas-Futures/data
    select: ".table-bigdata>tbody>tr>td:nth-child(2)"
    value_template: '{{ value | replace (".",",") | float }}'
    unit_of_measurement: "€/MWh"
    name: "Dutch TTF Gas"
    scan_interval: 900

Schermafbeelding 2022-10-18 om 14.44.07
Schermafbeelding 2022-10-18 om 14.44.07786×166 18.3 KB

I tried many times with other selectors copied but my sensor keeps “Unknown” in dutch “Onbekend”.

Can anybody help to come any further? HA restarting over and over gets annoying :wink:

Thanks in advance!

2

Unfortunately what you see in your web browser is not what the scrape integration sees. The raw HTML looks very different and essentially is just a SCRIPT tag with a huge JSON payload.

You can extract that JSON, but unfortunately it appears as if the payload is returned as an object, not a string, so I used the regex_replace filter to force this into a string. I’m not sure if that value you are looking for is directly available in the JSON payload, but as an example the following configuration just extracts the first attribute:

sensor:
- platform: scrape
  resource: https://www.theice.com/products/27996665/Dutch-TTF-Gas-Futures/data
  select: "body script"
  value_template: "{{ ((value | regex_replace(find='', replace='', ignorecase=False)) | from_json).pageId }}"
  unit_of_measurement: "€/MWh"
  name: "Dutch TTF Gas"
  scan_interval: 300
1 Like
3

Hi exxamalte,

Thanks, your configuration gives me some result now.

Schermafbeelding 2022-10-19 om 16.45.20
Schermafbeelding 2022-10-19 om 16.45.20786×280 32.6 KB

I think this is the price from one year ago, so I have to find out where today’s last price hangs out …

Thanks for helping me.

1 Like
4

Hi exxamalte,

I tried several days and made many attempts to find the latest actual price but until now I didn’t succeed.

Is there some way to extract the payload-object so I know which attribute I need for the actual price?

I hope I don’t bother you but I’m totaly stuck.

I would appreciate your help if you can, thanks you in advance.

5

Alright, let’s see…

Is this the value you are interested in?

gas-futures
gas-futures2044×550 46 KB

I found that the page loads another JSON payload from this URL: https://www.theice.com/marketdata/DelayedMarkets.shtml?getContractsAsJson=&productId=4331&hubId=7979

And this JSON does appear to contain some kind of end-of-the-month price prediction? The JSON payload is an array and its first entry is for November and it does indeed contain lastPrice : 109.5.

Maybe that helps, but I have no idea how those URL parameters work (like productId or hubId), and of they change or not.

6

That right, that’s the value/price where I’m looking for. By the way, I only want this once an hour (3600 seconds)
The future price NOV22 will end at October 31 and then it will start with DEC22 contract. I found also the JSON payload you found but it didn’t work out for me for some reason.

Thanks for your effort and have a nice evening.

7

Hi!

Did you ever find out how to scrape the actual Dutch TTF Gasprices?

Thanks

8

Hi all,
would appreciate if someone could help on this.
the scape integration has changed and now has UI along with a yaml but I can not configure it with the link posted above

https://www.theice.com/marketdata/DelayedMarkets.shtml?getContractsAsJson=&productId=4331&hubId=7979

9

hi, i tried scraping first but it is quite difficult to get it working. I found a solution with Rest
thanks to https://jsonpathfinder.com/
and https://www.ice.com/marketdata/DelayedMarkets.shtml?getContractsAsJson=&productId=4331&hubId=7979
and Help understanding RESTful / JSON & template sensors

  - platform: rest
    name: gasprijs
    value_template: "{{ value_json[0].lastPrice }}"
    resource: https://www.ice.com/marketdata/DelayedMarkets.shtml?getContractsAsJson=&productId=4331&hubId=7979
2 Likes