How do I extract items from my JSON Shopping List using REGEX?

galaxy_explorer · June 13, 2021, 9:54pm

I was somewhat surprised to learn the Shopping List integration does not provide a service to call the actual shopping list as it’s only visible through Lovelace or the associated card. This effectively means that it can’t be used in automations or notifications (seems like a What The Heck candidate to me!). So until this becomes available in the integration, I am requesting some help to make this possible.

So far I realised there is an undocumented API to call the shopping list through a RESTful service. I have this successfully working using the following YAML configuration which returns a JSON string with all shopping list items but with limitations…

YAML sensor configuration

# Shopping list
- platform: rest
  name: Grocery List
  headers:
    authorization: !secret shopping_list
    content-type: 'application/json'
  resource: http://<HA-IP>:8123/api/shopping_list
  value_template: '{{ value_json[0].name }}, {{ value_json[1].name }}, {{ value_json[2].name }}, {{ value_json[3].name }}'
  method: GET
  scan_interval: 60

Returned JSON string example

[
    {
        "name": "Milk",
        "id": "d08f74d8ea3d46cab50daed30386cd52",
        "complete": false
    },
    {
        "name": "Cereal",
        "id": "5f23907a1fc34fa4911aa5a7266f86a4",
        "complete": false
    },
    {
        "name": "Bread",
        "id": "2d258ae619fe4b17aaf404f108bba09d",
        "complete": false
    },
    {
        "name": "Fruit",
        "id": "518db2becf024d1ba157c618d452469f",
        "complete": false
    }
]

Example sensor results shown in entity alongside Shopping List Card

The first limitation is that I’ve hardcoded the number of shopping list items in the value template. So if there are fewer shopping list items than the stated number (4) it causes the RESTful service to fail and likewise if there are more shopping list items they are not captured. I need to create a for/loop statement of such to dynamically capture the number of items. Similar example here using the command sensor.

The second limitation is the template does not filter items that are “complete” and therefore all items are returned regardless of whether they were purchased or not. It would be nice just to return items that need to be bought.

Once the above limitations are sorted it should be straightforward to send the sensor state through Telegram message or any notification. I am requesting help from someone with more skill and experience in Jinja/templates, thanks.

Vasco · June 13, 2021, 10:34pm

I am very interested in this solution too!

galaxy_explorer · June 15, 2021, 9:52pm

After various searches it would seem that a regular expression (REGEX) is required to extract the data from the string. I have used a REGEX online tool to confirm the following expression should do the trick

"name": "([^"]+)",

REGEX results

Match 1
1.	Milk
Match 2
1.	Cereal
Match 3
1.	Bread
Match 4
1.	Meat
Match 5
1.	Fruit

However, I have tried to update the value template but it constantly errors about the syntax. Here is what I have so far

#  value_template: '{{ value | regex_findall_index("name": "([^"]+)",) }}'

I am requesting help to correct the above syntax and to confirm this is the best method / approach

galaxy_explorer · June 18, 2021, 4:55pm

@petro apologies for the direct tag, please feel welcome to ignore this message… I was hoping that someone could give me a direction on where I’m going wrong, I’ve put in the leg work researching this problem but it’s just that I’m stuck after playing around with REGEX for many hours…

petro · June 18, 2021, 4:59pm

You’re overcomplicating it. Next time, take a look at jinja’s api, which is what templates use. There’s alot of useful methods, like select_attr or map.

# Shopping list
- platform: rest
  name: Grocery List
  headers:
    authorization: !secret shopping_list
    content-type: 'application/json'
  resource: http://<HA-IP>:8123/api/shopping_list
  value_template: "{{ value_json | map(attribute='name') | list | join(', ') }}"
  method: GET
  scan_interval: 60

In regards to regex, this wouldn’t have even worked anyway you look at it. For some reason, regex_findall_index was implemented in home assistant with 2 huge glaring issues.

You have to specify an index, you can’t get a list of matches.
You can’t specify the matched group.

Because of this, you couldn’t get a list (like you want). And you can’t properly select what you really want, which is the value of the name. Proper regex for this would be:

{{ value | regex_findall_index('(?:"name": ").+(?:")', 0) }}

the (?: with a closing ) means “Non-capturing group”. Which means it would omit "name": " from the resulting match. Same goes for the closing ". However, regex matches always return the full match followed by the groups. Whoever implemented regex_findall_index defaulted the method to only return the full match and you aren’t able to specify a group which would give you just the name’s value.

Basically, these 2 “features” make regex_findall_index pretty useless in my opinion.

galaxy_explorer · June 19, 2021, 7:47pm

@petro thank you very much, that solved it and thanks for the direction for future reference

The detailed explanation of the REGEX limitations also explains why I was having so much trouble matching groups, I was messing around with this for hours, so partially glad that it wasn’t just me!

I do hope the REGEX function is enhanced as I gather it won’t be the last time, but more than happy with the above solution! Thank you.

ppistole · April 22, 2022, 6:56pm

how can i filter the completed and uncompleted items?

EDIT:

solution:

value_template: "{{ value_json | selectattr("complete", "false") | map(attribute='name') | list | join(', ') }}"

nb84 · August 4, 2022, 1:32pm

Sorry to necro this but I’m still really new to HA and I’m trying to setup an automation to put certain shopping_list items into a notification.

I’ve got the notification automation all set, tested, working, but it’s just hard coded text. Where I’m running into issues is implementing this sensor (nice idea btw). The config check is complaining that I don’t have a secret for shopping_list. I’m aware of where it needs to be created, but I’m not sure what the value needs to be, or how that’s determined.

galaxy_explorer · August 4, 2022, 2:05pm

You need to create a “Long-Lived Access Token” which is found under the Profile, scroll right to the bottom of the page to find the option. Create a new token called “Shopping List” and use the value in your secrets.yaml file or directly in the service call.

Capture

nb84 · August 4, 2022, 2:58pm

Awesome! Thank you!

tom_l · August 6, 2022, 6:43am

I only need to know if the list is empty so I can prevent a notification being sent when I enter a shopping zone with a condition. So I use this.

- resource: http://10.1.1.100:8123/api/shopping_list
  headers:
    authorization: !secret shopping_list
    content-type: 'application/json'
  method: GET
  scan_interval: 60
  binary_sensor:
  - name: Shopping List Empty
    value_template: >
      {% if value_json is defined %}
        {{ value_json|selectattr('complete', 'false')|map(attribute='name')|list|length == 0 }}"
      {% else %}
        false
      {% endif %}

Unfortunately the is defined check is required as otherwise the rest sensor generates an error at start up. Looks like home assistant starts before the API is ready or something.

It works perfectly after that.

manunited10 · August 28, 2023, 9:00am

Writing this here since it took me a while to figure out what I’m doing wrong.
If your Long-Lived Access Token is for example ABCDEFGHIJKL, in your secret file (or using directly in confiugration.yaml) add Bearer (note there’s a space) to it. This means our Example would be

# This is secrets.yaml
shopping_list: "Bearer ABCDEFGHIJKL"

galaxy_explorer · September 9, 2023, 7:39am

Here is a slightly updated version which fixes the log error when Home Assistant is restarted. It just checks whether the response is valid.

# Shopping list
- platform: rest
  name: Grocery List
  headers:
    authorization: !secret shopping_list
    content-type: 'application/json'
  resource: http://<your_id_address>:8123/api/shopping_list
  value_template: >
    {% if value_json is defined %}
      {{ value_json | map(attribute='name') | list | join(', ') }}
    {% else %}
      false
    {% endif %}
  method: GET
  scan_interval: 300

manunited10 · October 14, 2023, 12:52am

And finally let’s add truncate so if the shopping list is a bit long it doesn’t show unknown:

    value_template: "{{ value_json | selectattr('complete', 'false') | map(attribute='name') | list | join(', ') | truncate(255) }}"

kbrown01 · October 14, 2023, 1:54am

I would put the JSON in attributes (unless you shopping list is really short). Then you can format however you like and not be slave to 255 characters