Scrape the dynamic graph

Energy
,
1

how do I scrape a dynamic graph of the internet?
the url is:
https://mijn.nextenergy.nl/Website_CW/MarketPrices
i want to scrape the basic+ graph and display the whole graph in my home assistant (that’s what we pay)
can not get it to work… not even with chatgpt

2

image
image983×187 6.52 KB

Scraping does not have Javascript

3

so what in Home Assistant would be able to display this?

4

Do it properly

Not sure that is correct I just googled and that is what came up

5

If @Hellis81’s suggestion doesn’t get you anywhere, the data that makes up that graph is from here:

https://mijn.nextenergy.nl/Website_CW/screenservices/Website_CW/MainFlow/WB_EnergyPrices/DataActionGetDataPoints

…which I found using browser DevTools. Note that it’s a POST request with some parameters that you’ll have to duplicate. On my browser, these were:

{"versionInfo":{"moduleVersion":"5Hr44MwI_1DxLWhyy61m7A","apiVersion":"Bc8_XAfp3W7kn4dKB4nXdg"},"viewName":"MainFlow.MarketPrices","screenData":{"variables":{"Graphsize":348,"Filter_CurrentHour":8,"Filter_PriceDate":"2024-09-11","Filter_CostsLevelId":1,"IsOpenPopup":false,"HighchartsJSON":"","DistributionId":3,"IsDesktop":false,"IsTablet":false,"CurrentElectricityPrice_Local":"","CurrentElectricityPricePlus_Local":"","IsLoading":true,"NE_StartDate":"2022-07-01"}}}

That then returns:

{
    "versionInfo": {
        "hasModuleVersionChanged": false,
        "hasApiVersionChanged": false
    },
    "data": {
        "DataPoints": {
            "List": [
                {
                    "Label": "0",
                    "Value": "0.05",
                    "DataSeriesName": "Stroom",
                    "Tooltip": "0u €0,05",
                    "Color": "var(--color-orange-shade)"
                },
                {
                    "Label": "1",
                    "Value": "0.04",
                    "DataSeriesName": "Stroom",
                    "Tooltip": "1u €0,04",
                    "Color": "var(--color-orange-shade)"
                },
[…and so on]

You’d need to build a set of sensors using the RESTful integration to pull in the data, then create a graph to display it.

You’re wasting your time even trying.

1 Like
6

Ive got a similar question to the original one of this topic. Im trying to scrape/get some data into home assistant from the following link; https://energie.theoxygent.nl/api/prices_v2.php

Backstory, the data I am actually interested in is the forecasted energy price per hour for the next days from this website https://energie.theoxygent.nl/
It only appears upon hovering over the graph.
Not sure if it is easier/more doable ot get it out of that hovering objects or out of the link above.

I do have the calculations figured out how to extract that data from prices_v2.php.
I basically need the Y coordinates from the second block between [ and ] loaded into HA, and than I can calculate it from there.

Hope someone can help, thanks in advance!

7

https://energie.theoxygent.nl/api/prices_v2.php returns a JSON, so you can look up the values, but for any practical application, you would indeed need to know the X-axis reference. Do you know for which X reference you would need the Y value?

8

Yes, I know. I actually need multiple X references if possible cause I would love to know for example the next 48hours (maybe in something like an array).
The data is divided in 6 blocks in between [ ] brackets.
The first block seems to be the known values for the next day and are not of interest.
Any of the other 5 blocks are directly related to 1 sample for every hour, starting with the first sample being today at 12AM.
So I would like a array containing the first 48 Y values of the second block opened with [ bracket.

9

I found my solution! Had to look for a command line sensor to make this work.

1 Like
10

Great to hear! I was going to suggest a rest sensor:

rest:
. - resource: https://energie.theoxygent.nl/api/prices_v2.php
    scan_interval: 86400
    sensor:
      - name: "Priced"
        unique_id: prices
        value_template: {{value_json[0][0].y}}

You could make a loop to get more value by increasing the second 0 in the JSON dictionary.