Scrape HTTPS JavaScript generated content

Configuration
1

After nights researching I hope to find help here.
I want to scrape some values from a JavaScript-generated Website
https://main.beehivemonitoring.com/47c9f648af3b43d5ba07256925080639
Like “WEIGHT CHANGE 24H” from the tab “WEIGHT”.
I got no clue how to handle this, I tried so much…
Any hints?

2

You have all the data in json in https://main.beehivemonitoring.com/api/hives?culture=en-EN.
You have to add a X-Auth-Token header, though.

❯ curl -H "X-Auth-Token: 47c9f648af3b43d5ba07256925080639" "https://main.beehivemonitoring.com/api/hives?culture=en-EN"

[{"text":["Everything is all right",null,null,null],"id":6131,"name":"Volk 10","simExpirationDates":[],"emptyWeightKg":0.0,"vbatGw":4.202,"rssiGw":-87,"gsmSignal":-87,"weightToday":-0.10,"weightWeek":2.28,"weightMonth":6.05,"activeBees":6181,"bees":255,"motherStatus":255,"swarming":255,"status":255,"vbatOut":4.171,"tempOut":16.50,"humidityOut":62.0,"weight":45.61,"rssiOut":-80,"maxTempOut":16.50,"minTempOut":16.50,"lastWeight":45.61,"pressureGw":942.8,"pollen":false,"closed":false,"open":false,"eggs":false,"mother":false,"fetusOpened":false,"fetusClosed":false,"toCheck":false,"projectName":"Malte Jaurich","credit":0.0,"validTo":"9999-12-31T23:59:59.999999","upFreq":0.5,"canUseMyBees":false,"notificationTime":75600,"showOnlySmartCity":false}]
1 Like
3

So from there you need to use a Rest sensor:

Couple of examples, assuming the data structure above without the % on the end (that may cause some trouble if it’s real):

rest:
  - resource: https://main.beehivemonitoring.com/api/hives?culture=en-EN
    headers:
      X-Auth-Token: "47c9f648af3b43d5ba07256925080639"
    scan_interval: 600
    sensor:
      - name: "Weight today"
        value_template: "{{ value_json[0]['weightToday'] }}"
        unit_of_measurement: "kg"
      - name: "Active bees"
        value_template: "{{ value_json[0]['activeBees'] }}"
        unit_of_measurement: "bees"
1 Like
4

Nah, it’s not. C/p issue :wink:

1 Like
5

Hey, thats really really cool, it works! Thank you both so much.
How did you find out that the data exists in https://main.beehivemonitoring.com/api/hives?culture=en-EN ?

6

Fire up DevTools on your browser, usually F12. Go to the Network tab and reload the page. Look for resources of type xhr and find one that contains a response with the data in:

image
image1065×482 42.3 KB

2 Likes
7

How did you get to know the URL with /api ?
Now I try to read wind speed from a weather-station:
→ Holfuy: EINKORN, Wind und Wettermesswerte

So I find wind speed in mjso.php?k=737 in value “speed”.
But what’s the correct URL?

image
image1540×833 185 KB

I do not succeed with:
image

8

OK, I get closer to it.

https://holfuy.com/puget/mjso.php?k=737

is the link.
Got it by clicking right mousebutton and:

image
image665×608 34.9 KB

I tried:
image

But state is still unknown:

image
image864×768 44.2 KB

1 Like
9

For future requests, please:

  • never post code as screenshots
  • don’t re-use threads for unrelated requests

TIA :slight_smile:

Exact same works for me

rest:
  - resource: https://holfuy.com/puget/mjso.php?k=737
    scan_interval: 60
    sensor:
      - name: Test speed
        value_template: "{{ value_json['speed'] }}"
        unit_of_measurement: "km/h"

image
image657×504 25.3 KB

10

Sorry, yes I will :wink:

sensor.test_speed is always zero. I tried a lot but I don’t have a clue why the rest-request doesn’t fetch the right value from the json for “speed”. :frowning:

11

The URL is returning zero for speed. Perhaps it’s not very windy. Barometric pressure is high:

{
   "valid":"ok",
   "updated":"<b>34<\/b> sec. ago",
   "speed":0,
   "dir":314,
   "dir_str":"NW",
   "gust":0,
   "temperature":29.4,
   "wind_chill":29.4,
   "humidity":40.3,
   "dew_point":14.5,
   "cloud_base":"<b>2099<\/b><span style=\"font-size:0.6em;\">m AMSL<\/span>",
   "pressure":1012.2,
   "s":5,
   "avg_speed":0,
   "max_gust":0,
   "avg_dir":303,
   "avg_dir_str":"WNW",
   "dir_rate":0,
   "dir_tendency":"stable",
   "speed_rate":0,
   "speed_tendency":"calm",
   "daily_avg_speed":0,
   "daily_max_wind":6,
   "daily_max_temp":31.9,
   "daily_min_temp":15,
   "rain":"Last rain at Jun-09 Fri 12h"
}

I don’t know what the k parameter in the URL is. The full API is shown here:

http://api.holfuy.com/live/

EDIT: I think there’s something needed in the headers of the request. Having looked at the weather site, reloading the php resource now gives 13 for the wind speed.

Best bet is to get yourself an account on the API rather than trying to pull numbers by guesswork.

1 Like
12

Ok, thx, I sent a request for an API key for this weather-station. Thanks a lot for your efforts!!!