Mutiscrape on an API response that may contain a list within a JSON of unknown length

Configuration
1

Hi Everyone!

I’m trying to optimize my supply chain by logging the inventory levels, available units and store count for products in a few stores a few times a day and log these levels in a timeseries database like influxDB so that I can run some analyses in Grafana. This is for a pet project in my city.

Information for all products is accessible via an API.

I’m using multiscrape in the service tab of Developer tools to prototype my code before I move it to configurations.yaml. I’m able to get a response from the endpoint, but I’m stuck at how to process the response using templates so that I can store the response for each product in sensor entities in home assistant. I don’t know how many products the API would respond with since sometimes the store runs out.

Does anyone know the best way to achieve this? Ideally I don’t have to writer a python script and can use existing HACS or other add-ons.

Example code

service: multiscrape.scrape
data:
  name: Product Inventory Scrape
  resource: <API request + state parameters>
  sensor:
    - unique_id: products
      name: Products
      value_template: "{!{ value_json.hits.hits }!}"

Response looks something like this

{
  "took": 100,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "hits": {
    "total": 300, <- this number changes
    "max_score": 1,
    "hits": [
      {
        "_index": "something",
        "_type": "product",
        "_id": 327885,
        "_score": 1,
          "sku": 327885,
          "availableUnits": 94,
          "last_updated": "2024-06-10 20:10:01",
          "_regularPrice": 47.99,
          "isOrganic": false,
          "storeCount": 6,
        },
        {
          "_index": "something",
          "_type": "product",
          "_id": 327885,
          "_score": 1,
            "sku": 327885,
            "availableUnits": 123,
            "last_updated": "2024-06-10 20:10:01",
            "_regularPrice": 47.99,
            "isOrganic": false,
            "storeCount": 16
          },
          298 other products in hits list
        ]
      }
    }

Thank you

2

Use a Rest sensor (not multiscrape, which should be a last resort) and pull hits into an attribute.

See here for another option:

3

Hi @Troon , thanks for your response. I tried the rest sensor out and got an error because of the size of json being parsed.

My config file has:

rest:
  - resource: <some URL>
    scan_interval: 3600
    sensor:
      - name: "Products"
        value_template: "{{value_json['hits']['hits']}}"

I got this error in the logs:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1198, in _async_write_ha_state
    hass.states.async_set(
  File "/usr/src/homeassistant/homeassistant/core.py", line 2313, in async_set
    state = State(
            ^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 1786, in __init__
    validate_state(state)
  File "/usr/src/homeassistant/homeassistant/core.py", line 223, in validate_state
    raise InvalidStateError(
homeassistant.exceptions.InvalidStateError: Invalid state with length 13078400. State max length is 255 characters.
2024-06-10 23:43:55.267 ERROR (MainThread) [homeassistant.helpers.entity] Failed to set state for sensor.bcl_products, fall back to unknown
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1198, in _async_write_ha_state
    hass.states.async_set(
  File "/usr/src/homeassistant/homeassistant/core.py", line 2313, in async_set
    state = State(
            ^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 1786, in __init__
    validate_state(state)
  File "/usr/src/homeassistant/homeassistant/core.py", line 223, in validate_state
    raise InvalidStateError(
homeassistant.exceptions.InvalidStateError: Invalid state with length 13078400. State max length is 255 characters.
2024-06-10 23:43:55.268 ERROR (MainThread) [homeassistant.helpers.entity] Failed to set state for sensor.bcl_products, fall back to unknown
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1198, in _async_write_ha_state
    hass.states.async_set(
  File "/usr/src/homeassistant/homeassistant/core.py", line 2313, in async_set
    state = State(
            ^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 1786, in __init__
    validate_state(state)
  File "/usr/src/homeassistant/homeassistant/core.py", line 223, in validate_state
    raise InvalidStateError(
homeassistant.exceptions.InvalidStateError: Invalid state with length 13078400. State max length is 255 characters.
4

That’s why I said to pull hits into an attribute — not the state.

rest:
  - resource: <some URL>
    scan_interval: 3600
    sensor:
      - name: "Products"
        json_attributes:
          hits: "{{ value_json['hits']['hits'] }}"
        value_template: "{{ now() }}"
5

Thanks a lot @Troon! This worked.

Minor correction for anyone that reads this

above instead of attributes: use json_attributes:

1 Like
6

Yeah, sorry. Writing on my phone, corrected above. If you only want the inner hits list you can do this:

json_attributes_path: "$.hits"
json_attributes:
  - hits
1 Like