How can I scrape my data from a secure account?

Hi,
I would like to display my Mobile Phone Balance on an HA Dashboard.
But I am not sure on the best tools/steps to do this?

  • Firstly, I suspect I will need an automating script (bot - is that the correct term?) to log into my account.
  • Then, once the information is displayed, save and parse the page for the info I want.

I’m guessing the second step would use the in-built “scrape” integration?
But its step one that stumps me.

Please don’t suggest “use their API”. When asking this question of their Tech Support, I have concluded that they “don’t know their ar** from their elbow”?

Regards, Martin

I have two similar use cases which both require a login.

  1. I want to automatically update the mileage of my car in the company portal
  2. I want to know the lending period of my library

For both I checked the calls my browser did and try to recreate them in Home Assistant.

This was possible for the first use case, because they had some kind of API that was called by the browser. I had to call an API with user + password to get a token. This token could be used to call the actual API. I used curl and the shell command to do both calls and store the response. It worked quite well until they exchanged their login system. No it is broken until find time to look into it.

For my second use case I do not have a working solution yet. They do not offer any kind of visible API and there login form has hidden fields which change from time to time. I played around with Selenium to automate the login process and extract the data. This works quite well. I have a Python script which gives me the information, but I didn’t had the time to integrate it into Home Assistant.