Multiscrape Missing Field

I am using Multiscrape to get gas and water prices from my utility. All works well except for one field. If I look at the response in the response file the value is there, but for some reason Multiscrape doesn’t see it. I’m sure I am just missing something but can’t figure it out despite searching and trying different suggestions. Any help would be appreciated!

Config (the first three work just fine, it’s the last one “monthly unit charge” that does not work)

  scan_interval: 14400
  sensor:
    - unique_id: mud_gas_service_charge
      name: Utility - MUD - Gas Service Charge
      icon: mdi:currency-usd
      select: ".field-node--body > div:nth-child(1) > div:nth-child(1) > table:nth-child(13) > tbody:nth-child(1) > tr:nth-child(1) > td:nth-child(2)"
      value_template: "{{ value.replace('$', '') }}"
    - unique_id: mud_gas_infrastructure_charge
      name: Utility - MUD - Gas Infrastructure Charge
      icon: mdi:currency-usd
      select: ".field-node--body > div:nth-child(1) > div:nth-child(1) > table:nth-child(13) > tbody:nth-child(1) > tr:nth-child(2) > td:nth-child(2)"
      value_template: "{{ value.replace('$', '') }}"
    - unique_id: mud_gas_base_unit_charge
      name: Utility - MUD - Gas Base Unit Charge
      icon: mdi:currency-usd
      select: ".field-node--body > div:nth-child(1) > div:nth-child(1) > table:nth-child(13) > tbody:nth-child(1) > tr:nth-child(3) > td:nth-child(2)"
      value_template: "{{ value.replace('$', '0') }}"
    - unique_id: mud_gas_monthly_unit_charge
      name: Utility - MUD - Gas Monthly Unit Charge
      icon: mdi:currency-usd
      select: ".field-node--body > div:nth-child(1) > div:nth-child(1) > table:nth-child(30) > tbody:nth-child(3) > tr:nth-child(4) > td:nth-child(6)"
      value_template: "{{ value.replace('$', '') }}"
      force_update: true
  log_response: true
  #parser: html.parser
  headers:
    User-Agent: Mozilla/5.0

Field from Site:
image

Log

2022-04-10 08:22:24 DEBUG (MainThread) [custom_components.multiscrape.scraper] Scraper_noname_1 # Utility - MUD - Gas Monthly Unit Charge # Select selected tag: None
2022-04-10 08:22:24 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_1 # Utility - MUD - Gas Monthly Unit Charge # Exception selecting sensor data: 'NoneType' object has no attribute 'name'
HINT: Use debug logging and log_response for further investigation!
2022-04-10 08:22:24 ERROR (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_1 # Utility - MUD - Gas Monthly Unit Charge # Unable to extract data
2022-04-10 08:22:24 DEBUG (MainThread) [custom_components.multiscrape.sensor] Scraper_noname_1 # Utility - MUD - Gas Monthly Unit Charge # On-error, set value to None
2022-04-10 08:22:24 DEBUG (MainThread) [custom_components.multiscrape.entity] Scraper_noname_1 # Utility - MUD - Gas Monthly Unit Charge # Updated sensor and attributes, now adding to HA

Page Response
image

Sensor Value
image

The error message says that your selector is wrong, it does not select anything.
I don’t think tbody:nth-child(3) is correct. It should probably be tbody:nth-child(1). However, it seems like a table with multiple <tbody>s is valid, so it might be correct.
You should be able to debug your selector using your web browser’s developer tools.
If you need further help, please post the whole HTML document (in plain text, not as a screenshot).

1 Like

Oh shoot, sorry, I though I pasted that below the screenshot. I have tried using the inspection console on Firefox, Chrome/Edge and even used a couple different CSS selector extensions and can’t seem to figure this out. I am wondering if it might be a bug; or maybe the site is wonky somehow? I’ll admit, I am no expert but know enough to figure most things out but this has me baffled.

Since it exceed the character limit I put it on paste bin: https://pastebin.com/wE5SQEuD

EDIT: new pastebin link

Your pastebin link says:

Error, this is a private paste or is pending moderation. If this paste belongs to you, please login to Pastebin to view it.

1 Like

Ugh, I don’t know why pastebin hates me; this one should work, I tried it in a separate browser: https://ghostbin.com/T3Yg7

Well, your selector seems to be working (at least in Firefox). It worked even after I disabled JavaScript in my browser, so your problem shouldn’t be caused by that either.
I tried to make your selector a bit simpler:

table:nth-of-type(2) > tbody > tr:nth-of-type(4) > td:nth-of-type(6)

However, I highly doubt that this will fix it.
Sorry, I think I’m clueless at this point.
Have you tried setting log_response to true? What do the -soup.txt files contain? Or is that what you posted on ghostbin?

1 Like

Oh man, that did it! Thank you so much!!!

Just for my education, what made you think to remove the beginning pieces? I tried several different combinations, but I don’t know enough to know about those little details – if there is a more granular pattern I’m not aware of.

Thanks again!

Basically, I started from scratch and went for the simplest thing possible.
I want to select the second table in the document, a tbody in it, the 4th tr and the 6th td.
I have no idea why my selector works when yours doesn’t.

Interesting. I’ll have to remember to keep things simple when I run into this again. Appreciate the help!