Scrape HTTPS JavaScript generated content

After nights researching I hope to find help here.
I want to scrape some values from a JavaScript-generated Website
Like “WEIGHT CHANGE 24H” from the tab “WEIGHT”.
I got no clue how to handle this, I tried so much…
Any hints?

You have all the data in json in
You have to add a X-Auth-Token header, though.

❯ curl -H "X-Auth-Token: 47c9f648af3b43d5ba07256925080639" ""

[{"text":["Everything is all right",null,null,null],"id":6131,"name":"Volk 10","simExpirationDates":[],"emptyWeightKg":0.0,"vbatGw":4.202,"rssiGw":-87,"gsmSignal":-87,"weightToday":-0.10,"weightWeek":2.28,"weightMonth":6.05,"activeBees":6181,"bees":255,"motherStatus":255,"swarming":255,"status":255,"vbatOut":4.171,"tempOut":16.50,"humidityOut":62.0,"weight":45.61,"rssiOut":-80,"maxTempOut":16.50,"minTempOut":16.50,"lastWeight":45.61,"pressureGw":942.8,"pollen":false,"closed":false,"open":false,"eggs":false,"mother":false,"fetusOpened":false,"fetusClosed":false,"toCheck":false,"projectName":"Malte Jaurich","credit":0.0,"validTo":"9999-12-31T23:59:59.999999","upFreq":0.5,"canUseMyBees":false,"notificationTime":75600,"showOnlySmartCity":false}]

1 Like

So from there you need to use a Rest sensor:

Couple of examples, assuming the data structure above without the % on the end (that may cause some trouble if it’s real):

  - resource:
      X-Auth-Token: "47c9f648af3b43d5ba07256925080639"
    scan_interval: 600
      - name: "Weight today"
        value_template: "{{ value_json[0]['weightToday'] }}"
        unit_of_measurement: "kg"
      - name: "Active bees"
        value_template: "{{ value_json[0]['activeBees'] }}"
        unit_of_measurement: "bees"
1 Like

Nah, it’s not. C/p issue :wink:

1 Like

Hey, thats really really cool, it works! Thank you both so much.
How did you find out that the data exists in ?

Fire up DevTools on your browser, usually F12. Go to the Network tab and reload the page. Look for resources of type xhr and find one that contains a response with the data in:


How did you get to know the URL with /api ?
Now I try to read wind speed from a weather-station:
Holfuy: EINKORN, Wind und Wettermesswerte

So I find wind speed in mjso.php?k=737 in value “speed”.
But what’s the correct URL?

I do not succeed with:

OK, I get closer to it.

is the link.
Got it by clicking right mousebutton and:

I tried:

But state is still unknown:

1 Like

For future requests, please:

  • never post code as screenshots
  • don’t re-use threads for unrelated requests

TIA :slight_smile:

Exact same works for me

  - resource:
    scan_interval: 60
      - name: Test speed
        value_template: "{{ value_json['speed'] }}"
        unit_of_measurement: "km/h"

Sorry, yes I will :wink:

sensor.test_speed is always zero. I tried a lot but I don’t have a clue why the rest-request doesn’t fetch the right value from the json for “speed”. :frowning:

The URL is returning zero for speed. Perhaps it’s not very windy. Barometric pressure is high:

   "updated":"<b>34<\/b> sec. ago",
   "cloud_base":"<b>2099<\/b><span style=\"font-size:0.6em;\">m AMSL<\/span>",
   "rain":"Last rain at Jun-09 Fri 12h"

I don’t know what the k parameter in the URL is. The full API is shown here:

EDIT: I think there’s something needed in the headers of the request. Having looked at the weather site, reloading the php resource now gives 13 for the wind speed.

Best bet is to get yourself an account on the API rather than trying to pull numbers by guesswork.

1 Like

Ok, thx, I sent a request for an API key for this weather-station. Thanks a lot for your efforts!!!