Help with using Multiscrape with Form-submit to log into Lose It!

Configuration
1

Heyoo, long time user, first time poster (please go easy on me).

I am trying to use the Multiscrape HACS integration to pull the calories I have remaining in the day from Lose It! because there is no Lose It! API available for personal projects.

Lose It! has a login form here - Lose It!

This is my Multiscrape configuration.yaml code:

multiscrape:
  - name: Scraper McScrapeface
    resource: 'https://www.loseit.com/' #method is not defined and defaults to GET
    scan_interval: 36000
    log_response: True
    form_submit:
      submit_once: True
      resource: 'https://my.loseit.com/login'
      input:
        email: !secret LoseIt_username
        password: !secret LoseIt_password
    sensor:
      - unique_id: calories_remaining
        name: Calories Remaining
        select: "#applicationPanel > div > div > div.GCJ-IGUNGC > div > table > tbody > tr:nth-child(3) > td > table > tbody > tr > td:nth-child(2) > table > tbody > tr:nth-child(1) > td > table > tbody > tr:nth-child(2) > td > div > div > table > tbody > tr:nth-child(3) > td > table > tbody > tr > td > div > strong"

And this is the debug log:

2024-01-31 16:14:34.344 WARNING (SyncWorker_2) [homeassistant.loader] We found a custom integration multiscrape which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-01-31 16:14:44.652 ERROR (MainThread) [custom_components.multiscrape.coordinator] Scraper McScrapeface # Exception in form-submit feature. Will continue trying to scrape target page.
Client error '405 Method Not Allowed' for url 'https://my.loseit.com/login'
For more information check: https://httpstatuses.com/405
2024-01-31 16:14:45.492 ERROR (MainThread) [custom_components.multiscrape.sensor] Scraper McScrapeface # Calories Remaining # Unable to scrape data: Could not find a tag for given selector 
Consider using debug logging and log_response for further investigation.

I went down a rabbit hole of headers, I’m not 100% sure what I was looking at but that could be a source of the problem.

I’m running Home Assistant v2023.9.3 on a Windows VirtualBox virtual machine.

Thanks in advance if you see this and know what’s up.

2

Use the log_response option as documented in the multiscrape docs to see what the server response is, and then post it here. Perhaps you just need to change the method from GET to POST.

3

I ended up following this post:

to get the value because the page is dynamic. Very good guide.

4

Do you mind sharing your yaml code from what you ended up learning? I am heading down this path and if someone has already invented the wheel . . . Many thanks!

5

The following makes sense in the context of this guide.

My configuration.yaml includes:

shell_command:
  browserless_scraper: "bash ./scripts/browserless_scraper.sh {{function}} {{output}}"

multiscrape:
    - name: Scraper McScrapeface
      resource: "http://localhost:8123/local/browserless/my_output.html"
      scan_interval: 0
      timeout: 30
      log_response: true
      method: GET
      sensor:
        - name: Calories Remaining
          unique_id: calories_remaining
          select: "div > table > tbody > tr:nth-child(3) > td > table > tbody > tr > td:nth-child(2) > table > tbody > tr:nth-child(1) > td > table > tbody > tr:nth-child(2) > td > div > div > table > tbody > tr:nth-child(3) > td > table > tbody > tr > td > div"

My browserless_scraper.sh looks like:

#!/bin/sh
curl -X POST 'http://homeassistant.local:3000/function' --data-binary '@/config/js_scrapers/'$1 -H 'Content-Type: application/javascript' > /config/www/browserless/$2

My automation looks like:

alias: When triggered, update Calories remaining
description: ""
triggers: []
conditions: []
actions:
  - data:
      function: my_scraper.js
      output: my_output.html
    action: shell_command.browserless_scraper
    alias: Scrapes HTML into /homeassistant/www/browserless/my_output.html
  - delay:
      hours: 0
      minutes: 0
      seconds: 2
      milliseconds: 0
  - data: {}
    action: multiscrape.trigger_scraper_mcscrapeface
    alias: >-
      Reads my_output.html and updates sensor.calories_left in
      /homeassistant/configuration.yaml
mode: single

My my_scraper.js looks like this (I use http://homeassistant.local:3000/debugger/ for running in-browser debugging):

export default async ({ page }) => {
  await page.goto("https://my.loseit.com/login");
  await page.type('input[id*="email"]', "YOUR EMAIL GOES HERE");
  await page.type(
    'input[id*="password"]',
    "YOUR PASSWORD GOES HERE"
  );

  await Promise.all([
    page.keyboard.press('Enter'),
    page.waitForNavigation(),
  ]);

  await Promise.all([
    page.goto("https://www.loseit.com/#Home"),
    page.waitForSelector("div > table > tbody > tr:nth-child(3) > td > table > tbody > tr > td:nth-child(2) > table > tbody > tr:nth-child(1) > td > table > tbody > tr:nth-child(2) > td > div > div > table > tbody > tr:nth-child(3) > td > table > tbody > tr > td > div"),
  ]);

  const html = await page.content();

  return html;
};