Using Selenium with chromedriver

Hi there,

I have a python script that uses Selenium and Chrome Webdriver to log in to my bank website to scrape the account balances.

It’s working as expected in windows, I’d now like to run it from within Home Assistant once or twice daily, and have the values update relevant HA sensors.

I was thinking that Appdaemon would be the simplest way to run the script, so I installed the Appdaemon addon and configured Selenium package to be installed. But I can’t figure out how/where to install Chromedriver and how to specify its location.

My system details:

 Appdaemon add-on version: 0.3.2
 System: Ubuntu 18.04.5 LTS  (amd64 / qemux86-64) on Hyper V
 Home Assistant Core: 2020.12.1
 Home Assistant Supervisor: 2020.12.7

The errors I see in the Appdaemon log when loading my app.py:

FileNotFoundError: [Errno 2] No such file or directory: 'chromedriver'

and

selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH.

Could somebody please point me to an example where Selenium & webdriver is being used with Home Assistant, as my searches have not fared so well?
The other option I was thinking was to run the python app on another VM, outside of Home Assistant, with the values being sent via MQTT. But this feels like a bit of a waste of resources.

cheers, Anthony

1 Like

Did you get this sorted out?

Sorry I just saw your message now. I ended up going the route of a separate VM to run the python/selenium code, which then sends the value to HA via MQTT. Crude but it works.

Thanks - I landed on the same.

I successfully run a python script with Selenium and Chrome Webdriver in Appdaemon of Hassio for 5 hour debuging.
Here are the key setting:
the setting in AppDaemon4 of Hassio:

system_packages:
  - chromium-chromedriver
  - chromium
python_packages:
  - selenium

the code in python:

        from selenium import webdriver
        from selenium.webdriver.chrome.options import Options
       
        chrome_options = Options()
        chrome_options.add_argument('--headless')
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-gpu')
        chrome_options.add_argument('--disable-dev-shm-usage')
        driver = webdriver.Chrome('chromedriver', options=chrome_options)

good luck!

6 Likes

I run selenium with apps in Appdaemon. This is the helper file I import into appdaemon apps to help me run scripts. I’m still working on it so it is a mess ,but it works.

For selenium I run it as a docker image in hub mode using this image.

You can install the selenium driver by adding selenium to your requirements.txt file.

1 Like

I run my hassio in a VM on a win10 pc, is there a way to make this work? I presently scrape using Excel and VBA so wanted to know if I can mimic this and have sensors within Hassio record my scraping.

This is my py code:

import logging
import time
CONF_ID = "id"
CONF_ACCOUNT_KEY = "account_key"
CONF_NAME_OVERRIDE = "name_override"

SENSOR_PREFIX = "Kadenabalance "

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
     
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver', options=chrome_options)

driver.get("https://balance.chainweb.com/index.html")
time.sleep(5)
driver.find_element_by_id("account").SendKeys (CONF_ACCOUNT_KEY)
driver.find_element_by_class_name("button").Click
time.sleep(5)
ATTR_BALANCE = driver.find_element_by_id("chain-0-balance-data").Text

_LOGGER = logging.getLogger(__name__)

I keep getting: Platform error sensor.kadenabalance - No module named 'selenium’

I added this to AppDaemon 4:

init_commands: []
python_packages:
  - selenium
system_packages:
  - chromium-chromedriver
  - chromium

Hi, sorry for reviving this topic!

I tried this code but returned the same error of executable needs to be in PATH.
So, I put the path in the code but now the error is
OSError: [Errno 8] Exec format error: ‘/config/chromedriver’
I downloaded the chromedriver and put it in the config folder… not sure if the version of chromedriver or some permission problem…
What would be the path of chromedriver installed by AppDaemon?
I tried other packages too, like chromedriver-manager but no luck to HA recognize the import, even with AppDaemons showing a successful installation…
Anyway… thanks in advance for any help!
Regards

Have a look at GitHub - mdeweerd/veolia-idf: Charge l'historique de votre consommation Veolia Ile de France - it’s in French because this is for a french water supplier, but you can translate if needed an see the configuration in plain yaml anyway.

You can checkout veolia-idf/veolia-idf-domoticz.py at e2a6b760983c2620fb1f3721e452d937d6173ec8 · mdeweerd/veolia-idf · GitHub to see how the location is determined automatically.

1 Like

Hi everyone,
I think I have solved it!
First write these commands in Appdaemon configuration:

echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" > /etc/apk/repositories
echo "http://dl-cdn.alpinelinux.org/alpine/edge/main" >> /etc/apk/repositories
apk update

Then specify all the python packages you need, like selenium.

And then also specify the following Alpine Linux packages:

  • chromium
  • chromium-chromedriver

My configuration looks like this:

Finally, my code to create the driver was:

options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')
options.add_argument('--disable-dev-shm-usage')

driver = webdriver.Chrome('/usr/bin/chromedriver', options=options)

I have not figured out if ALL the steps and packages are actually required but I wanted to share this as soon as it worked for me. Please let me know if this solve the problem for you!

Sources:

1 Like

The packages that I added through the AppDaemon configuration are like this - I found no need to extend the repositories - you may have because you included less “binary” packages.

init_commands: []
python_packages:
  - selenium
  - PyVirtualDisplay
system_packages:
  - py-urllib3
  - py3-colorama
  - xvfb
  - py3-pip
  - xorg-server-xephyr
  - chromium-chromedriver
  - chromium
  - py3-openssl
  - py3-pysocks
  - py3-wsproto
  - py3-sniffio
  - py3-async_generator
  - py3-sortedcontainers
  - py3-attrs
  - py3-outcome
  - py3-trio

And the chromedriver setup is a bit more complex: - you may want to pickup some options MetersToHA/meters_to_ha.py at 4782b5762f636f7be86436d957d6edee87512cf6 · mdeweerd/MetersToHA · GitHub .

1 Like

I used this comment a while ago to get python scripts working for third party imports. I was using AppDaemon to install the requirements and system packages, and the system python interpreter. It seems like it’s no longer working. Could anyone else confirm?

I made a detailed post about it on reddit. Anyone know a solution?
https://www.reddit.com/r/homeassistant/comments/147a8a7/python_and_appdaemon/

Its not clear what is no longer working, but I recommend not setting absolute version requirements on the packets as that evolves over time. You can use “>=” or no version instead.

Setting absolute or loose makes no difference.

All I can determine is that my python script is failing to import selenium. I determined this by directing the “output” to a file.

- switch:
    unique_id: tp_link_reboot
    name: TP Link Router Reboot
    command_off: "python3 /config/scripts/python/tp_link_reboot.py --my_args > /config/scripts/python/tp_link_debug.log 2>&1"
    command_state: 'curl "http://192.168.1.1/cgi-bin/luci/;stok=/locale?form=lang" -X POST --data-raw "operation=read" | grep -oF true'
    value_template: '{{ value == "true" }}'

output

Traceback (most recent call last):
  File "/config/scripts/python/tp_link_reboot.py", line 8, in <module>
    from selenium import webdriver
ModuleNotFoundError: No module named 'selenium'

AppDaemon config

system_packages:
  - chromium-chromedriver
  - chromium
python_packages:
  - selenium
  - requests
init_commands: []

Doesn’t seem to be picking up the AppDaemon dependencies/environment as it used to

Rolling back AppDaemon did not help.

If you are running the script in the home assistant docker container where these packages are not installed, as the configuration is valid for the appdaemon container.

AppDaemon is a docker container on its own and scripts must be run from within appdaemon and that is set up in the ‘/config/appdaemon’ directory tree where you would normally be keeping the scripts as well.

To call a script, you should be able to use my setup:

/config/appdaemon/apps/call-script/call_script.py:

import os
import sys
import subprocess as s

import adbase as ad
import hassapi as hass


class CallScript(hass.Hass):
    def initialize(self):
        event_name = "call_script"
        if "event" in self.args:
            event_name = self.args["event"]
        self.listen_event(self.call_script, event_name)
        self.call_script()

    @ad.app_lock
    def call_script(self, *args, **kwargs):  # pylint: disable=unused-argument
        if "script" not in self.args:
            raise("Missing script parameter for CallScript")

        script_args = [self.args["script"]]
        if "args" in self.args:
             s_args = self.args["args"]
             script_args.extend(s_args)

        out = sys.stdout
        err = sys.stderr
        closeout = False
        closeerr = False

        if "outfile" in self.args:
            out =  open(self.args["outfile"], "w")
            closeout = True
        if "errfile" in self.args:
            err =  open(self.args["errfile"], "w")
            closeerr = True

        s.call(script_args, stdout=out, stderr=err)
        
        # Close files as needed
        if closeout:
             out.close()
        if closeerr:
             err.close()

Then add a configuration in apps, example: /config/appdaemon/apps/scripts.yaml:

chrome_filetype:
   module: call_script
   class: CallScript
   event: call_file
   script: /usr/bin/file
   args:
     - /usr/bin/chromedriver
     - /usr/bin/chromium-browser
     - /usr/bin/python3
     - /usr/bin/python3.10
     - /usr/lib/chromium/chromedriver
   outfile: /config/appdaemon/apps/script.log
   errfile: /config/appdaemon/apps/scripterr.log

The above will call ‘/usr/bin/file’ in the AppDaemon container when the ‘call_file’ event is triggered.

In my case that results in /config/appdaemon/apps/script.log containing:

/usr/bin/chromedriver:          symbolic link to ../lib/chromium/chromedriver
/usr/bin/chromium-browser:      symbolic link to ../lib/chromium/chromium-launcher.sh
/usr/bin/python3:               symbolic link to python3.11
/usr/bin/python3.10:            cannot open `/usr/bin/python3.10' (No such file or directory)
/usr/lib/chromium/chromedriver: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, BuildID[sha1]=1e63da5a822ad402ffc12658fe24f2939b40daba, stripped

Adding a configuration file ‘/config/appdaemon/apps/tplink.yaml’ as follows with ‘call-script’ in place should execute your script with access to the selenium package as expected when firing the event ‘reboot_tplink’:

tplink:
   module: call_script
   class: CallScript
   event: reboot_tplink
   script: /usr/bin/python3
   args: 
     - /config/scripts/python/tp_link_reboot.py
     - --args
   outfile: /config/scripts/python/tp_link_debug.log
   errfile: /config/scripts/python/tp_link_debug_err.log
The chromedriver version (115.0.5790.170) detected in PATH at /usr/bin/chromedriver might not be compatible with the detected chrome version (116.0.5845.96); currently, chromedriver 116.0.5845.96 is recommended for chrome 116.*, so it is advised to delete the driver in PATH and retry

Shouldn’t it update every time there’s a new version?

I’m trying to set up an app using selenium and I’m using selenium 4.23.0 AppDaemon is configured to add chromium and chromium-chromedriver for system packages and selenium as well for python. I’ve verified that chromium and chromedriver are installed via the script, and they’re both in /usr/bin, which is in the PATH. If I fire up a docker container with the same setup, my script works fine. In AppDaemon, however, it says it can’t find the driver. Any ideas?