Web scraping

Configuration
1

I am a complete novice on how to select what I want out of a web page using the web scraper sensor. Can someone help.
I am trying to find if my router is in mobile mode. I need therefore to see what the class is in ledMobile.
What do I ‘select’ and what do I do to read back the value in the class.?
Thanks

router
router712×487 29.5 KB

2

Maybe I’m wrong but I don’t think you can get that.
That class value you are looking for is part of the html tag, and I believe the scraper only gets html “values” between tags.

If that is correct then I think you will need to use regular expressions to get the value.
I believe someone posted a regular expressions tool for scraper some time ago

3

Have a go at scraping for:

select: "span[name='ledMobile'].off-light"

No idea if that’ll work, but hopefully it’ll return something different whether off or on. Perhaps unavailable or blank? Then you could create a template sensor from it.

4

Just get ‘unknown’ state. :confounded:

5

Found a way to a solution

- platform: command_line
  command: python3 -c "import requests; print(requests.get('http://192.168.0.1/login.htm').text.split('name=\"ledMobile\"',1)[1][12:16])"
  name: NBN Mobile Mode
  scan_interval: 300
2 Likes
6

Nice :+1:

To those wondering what this alchemy does, look at the HTML above. This reads the HTML all in, splits it at the name="ledMobile" bit, takes the part after the split, and returns the 13th to 16th characters (the “12” index starts at zero). This is " off" or " gre" depending on status (if I’ve counted correctly).

2 Likes
7

Unfortunately, under test I found this doesn’t work because the web page is driven by a bunch of front-end java script which doesn’t get executed when I do a requests.get. This means I always get the default value which is off. Back to the drawing board.
Any suggestions @Troon ??

8

Does the router have a different webpage for mobiles?
Sometimes people make a pure html version for mobiles and use the javascript versions for computer

9

great idea @Hellis81, but no. :grimacing:

10

Bugger…
Do you have a static IP normally?
Maybe that is a way to do it? Usually routers display the IP they have so if that change then you are on the 4G network.

11

hmm @Hellis81 that might do. Ping sensor doesn’t return the ip. Did you have a thought on how to do this?

12

@Hellis81’s idea of looking for visible changes in the network configuration is good, but without being able to see it ourselves, it’s hard to advise.

Presumably this is a Telstra router? Looks to me like it refreshes its LEDs via a call to a CGI script like this:

https://192.168.0.1/cgi/cgi_get_led_rear.js?_tn=000000000&_t=1593329622679

The t parameter is just a timestamp, but the tn seems to be an access token and may stop you accessing it from elsewhere (the nine-zeros above is a placeholder).

An alternative hardware solution would be to set something up looking at the LED like a photodiode and an ESP8266 module, but that’s probably a bit extreme.

13

There are services that display your public IP, but I believe they will block you if you keep spamming their server with requests.
I made a little webpage that we can test on for now.
Set up a scraper and select ‘p’ and you should get your public IP.
on this page:

http://www.hoppvader.nu/app/IP.php

The php code that runs here is as simple as:

<?php

    echo "<p>" . $_SERVER['REMOTE_ADDR'] . "</p>";
14

Cracked it for your idea @Hellis81 . Let’s see how it goes.

sensor:
  - platform: dnsip

automation:
  alias: Notify when external ip changes
  initial_state: on 
  trigger:
    platform: state 
    entity_id: sensor.myip
  action:
    service: notify.pushover
    data_template:
      message: "New external IP address is {{ states('sensor.myip') }}"
15

Thanks for your efforts @Hellis81 , I found that the dnsip platform has it built-in as a default to return your own ip.

16

It’s Telstra branded but called a ‘Arcadyan LH1000’. @Troon Your code looks pretty right but needs more skill than I to develop a hack.
Not going to the photo diode solution, much as I’d enjoy the project - just a commitment too far. Covid hasn’t got me that that locked down !

17

I didn’t know there was a dnsIP in HA.
Good that it worked. Does the IP switch when you unplug the cable?

Let’s just hope you have a static IP then :slight_smile: