Scraping actual price Dutch TTF Gas

I tried to scrape the actual gasprice form the Dutch TTF Gasprices but get stuck with it.

I would appreciate if someone can give me a solution to solve my problem.

My YAML looks like this

  - platform: scrape
    resource: https://www.theice.com/products/27996665/Dutch-TTF-Gas-Futures/data
    select: ".table-bigdata>tbody>tr>td:nth-child(2)"
    value_template: '{{ value | replace (".",",") | float }}'
    unit_of_measurement: "€/MWh"
    name: "Dutch TTF Gas"
    scan_interval: 900

I tried many times with other selectors copied but my sensor keeps “Unknown” in dutch “Onbekend”.

Can anybody help to come any further? HA restarting over and over gets annoying :wink:

Thanks in advance!

Unfortunately what you see in your web browser is not what the scrape integration sees. The raw HTML looks very different and essentially is just a SCRIPT tag with a huge JSON payload.

You can extract that JSON, but unfortunately it appears as if the payload is returned as an object, not a string, so I used the regex_replace filter to force this into a string. I’m not sure if that value you are looking for is directly available in the JSON payload, but as an example the following configuration just extracts the first attribute:

sensor:
- platform: scrape
  resource: https://www.theice.com/products/27996665/Dutch-TTF-Gas-Futures/data
  select: "body script"
  value_template: "{{ ((value | regex_replace(find='', replace='', ignorecase=False)) | from_json).pageId }}"
  unit_of_measurement: "€/MWh"
  name: "Dutch TTF Gas"
  scan_interval: 300
1 Like

Hi exxamalte,

Thanks, your configuration gives me some result now.

I think this is the price from one year ago, so I have to find out where today’s last price hangs out …

Thanks for helping me.

1 Like

Hi exxamalte,

I tried several days and made many attempts to find the latest actual price but until now I didn’t succeed.

Is there some way to extract the payload-object so I know which attribute I need for the actual price?

I hope I don’t bother you but I’m totaly stuck.

I would appreciate your help if you can, thanks you in advance.

Alright, let’s see…

Is this the value you are interested in?

I found that the page loads another JSON payload from this URL: https://www.theice.com/marketdata/DelayedMarkets.shtml?getContractsAsJson=&productId=4331&hubId=7979

And this JSON does appear to contain some kind of end-of-the-month price prediction? The JSON payload is an array and its first entry is for November and it does indeed contain lastPrice : 109.5.

Maybe that helps, but I have no idea how those URL parameters work (like productId or hubId), and of they change or not.

That right, that’s the value/price where I’m looking for. By the way, I only want this once an hour (3600 seconds)
The future price NOV22 will end at October 31 and then it will start with DEC22 contract. I found also the JSON payload you found but it didn’t work out for me for some reason.

Thanks for your effort and have a nice evening.

Hi!

Did you ever find out how to scrape the actual Dutch TTF Gasprices?

Thanks

Hi all,
would appreciate if someone could help on this.
the scape integration has changed and now has UI along with a yaml but I can not configure it with the link posted above

https://www.theice.com/marketdata/DelayedMarkets.shtml?getContractsAsJson=&productId=4331&hubId=7979

hi, i tried scraping first but it is quite difficult to get it working. I found a solution with Rest
thanks to https://jsonpathfinder.com/
and https://www.ice.com/marketdata/DelayedMarkets.shtml?getContractsAsJson=&productId=4331&hubId=7979
and Help understanding RESTful / JSON & template sensors

  - platform: rest
    name: gasprijs
    value_template: "{{ value_json[0].lastPrice }}"
    resource: https://www.ice.com/marketdata/DelayedMarkets.shtml?getContractsAsJson=&productId=4331&hubId=7979

2 Likes