Sort list by variable attribute/sort order in jinja

I’m working on integration of my movies library into HA. I have script runnin on my NAS that outputs list of available movies with selected attributes and TV shows into large json file. Now I want to display this information in UI, but it impractical as is, due to size of library. So I want to limit what is displayed to 100 entries per page + filter content by library (in my case HD or SD or Music or TV) and/or genre and then apply sort order that would make searching for specific movie easier. Sorting would need to be applied by some attributes (movie original title, Polish title, releaase date, IMDB rating etc. when movie was added to library and would be done in ascending or descending order. And sorting is what troubles me, as I can’t make it configurable. So what I’m seeking for is how to dynamically replace these sort attribute and order in follwong code:

{% set movies_all = state_attr('sensor.movies_list', 'list') | sort(attribute='Release Year', reverse=false) %}

I tried simple replacemnt with states of corresponding helpers:

{% set movies_all = state_attr('sensor.movies_list', 'list') | sort(attribute=states('input_select.sort_by'), reverse=states('input_boolean.sort_order')) %}

or to use variables for this:

{% set attr=states('input_select.sort_by')|string %} # this contains name off attribute to sort by
{% set ord=is_state('input_boolean.sort_order', 'on')|string %} # this should change the order of sorting from nnormal to reverse
{% set movies_all = state_attr('sensor.movies_list', 'list') | sort(attribute='attr', reverse='ord') %}

but neither of these works (OK, I understand that input_boolean gives on/off states instead of true/false, but this is for fixing later). At the moment the only solution I can think off that would work is to check the state of these 2 helpers using if elseif else conditions and put proper sorting command inside, but this would create 10 different cases (5 attributes * 2 sort orders)… not elegant solution for sure.
Any idea how to implement it properly?

Copy-paste the following version into the Template Editor and see if it produces the desired result.

{% set attr = states('input_select.sort_by') %}
{% set ord = is_state('input_boolean.sort_order', 'on') %}
{% set movies_all = state_attr('sensor.movies_list', 'list') | sort(attribute=attr, reverse=ord) %}

Be advised that when you wrapped attr in quotes, you converted it from being the name of a variable to a literal string.

attribute='attr'

I don’t know if the template you tested has this in it or you just added it for the version you posted.

# this contains name off attribute to sort by

That’s a YAML comment and it should not be included within a Jinja2 template.

This is a Jinja2 comment:

{# this contains name off attribute to sort by #}

I tested both, with and without the quotes, but neither of them worked.

I know, I added this to what I posted to add some explanation… and without thinking that it will create ‘syntax confusion’… In real life template does not contain these quotes.

BUT, what it means to have a fresh look… the problem was that I was trying various solutions at the same time, not necessary in consistant way. So, using variables (as in your sample, without quotes) and adding ord variable exactly as in this sample (previously I was usng variable directly with helper state (!) solved the problem… So I Was very close, but your comments pushed me to the right direction!

Are you saying the example I posted did not work?

What was the error it reported?

Or it did work? (I can’t be sure from your reply; you marked your own post as the Solution).

It did work! My problem was that at the time I originally tested, I messed up different stages of coming to final solution and that’s why I failed. You spoted all proper solutiins in one place.

Seems I’m still in the rush :man_facepalming:

1 Like

Hmm… seems I was to quick to announce success… upon careful checking of the results I found strange issue with sorting. It work fine for all attrubutes, but one Release Year. So, here is the final code of sensor (I know it could be done differenctly, as we discussed this here, but I found some other issues, which I will describe later):

template:
  - trigger:
    - platform: state
      entity_id: input_select.library
    - platform: state
      entity_id: input_select.sort_by
    - platform: state
      entity_id: input_number.movies_page
    - platform: state
      entity_id: input_boolean.sort_order
    sensor:
      - name: movies_list_fltr
        state: 'on'
        attributes:
          details: >     
            {% set attr = states('input_select.sort_by') %}
            {% set ord = is_state('input_boolean.sort_order', 'on') %}

            {% set movies_all = state_attr('sensor.movies_list', 'list') | sort(attribute=attr, reverse=ord) %}
            {% set movies_fltr = namespace(movies = '') %}
            {% set counter = namespace(i = 1) %}
            {% set json_string = '' %}
            {%- for movie in movies_all -%}
            {% if movie.Library == states('input_select.library') %}
              {% if counter.i >= (states('input_number.movies_page') | int * 100 - 99) and counter.i < (states('input_number.movies_page') | int * 100 + 1) %}
                {%- set imdbID = '{"imdbID":"' + movie.imdbID + '",' -%}
                {%- set Title = '"Original Title":"' + movie['Original Title'] + '",' -%}
                {%- set PLTitle = '"Polish Title":"' + movie['Polish Title'] + '",' -%}
                {%- set imdbLink = '"imdbLink":"' + movie['imdbLink'] + '",' -%}
                {%- set Rating = '"Rating":"' + movie['Rating'] + '",' -%}
                {%- set Genre = '"Genre":"' + movie['Genre'] + '",' -%}
                {%- set ReleaseYear = '"Release Year":"' + movie['ReleaseYear'] + '",' -%}
                {%- set PblDate = '"Publish Date":"' + movie['Publish Date'] + '",' -%}
                {%- set json_string = json_string+ imdbID + Title + PLTitle + imdbLink + Rating + Genre + ReleaseYear + PblDate +'},' -%}
                {% set movies_fltr.movies = movies_fltr.movies + json_string %}
              {% endif %}
            {% set counter.i = counter.i + 1 %}
            {% endif %}
            {% endfor %}
            {{ movies_fltr.movies }}

and definitions of corresponding helpers:

input_number:
  movies_page: # controls 100 elements page to be displayed
    name: Movies Page to Display
    icon: mdi:book-outline
    min: 1
    max: 1000

input_select:
  library: # selects library to limit number of objects to be parsed
    name: Movies Library
    options:
      - HD Movies
      - SD Movies
      - Music Movies
      - TV Shows

  sort_by: # controls by which attribute to sort the list
    name: Sort by
    options:
      - Publish Date
      - Release Year
      - Original Title
      - Polish Title
      - Rating

input_boolean:
  sort_order: # controls sort order (ascending/descending)
    name: Movies Attribute Sort Order
    icon: mdi:sort
    initial: off

And here is sample value of atribute being processed by this template:

details: 
- imdbID: aa0000001
  Original Title: David Garrett - Rock Symphonies
  Polish Title: "David Garrett: Na żywo, na świeżym powietrzu"
  imdbLink: http://www.imdb.com/title/aa0000001/
  Rating: N/A
  Genre: Music
  Release Year: '2010'
  Publish Date: '2012:07:27:11:06:02'
- imdbID: aa0000012
  Original Title: Jean Michel Jarre - Welcome to the other side
  Polish Title: Jean Michel Jarre - Welcome to the other side
  imdbLink: http://www.imdb.com/title/aa0000012/
  Rating: N/A
  Genre: Music
  Release Year: '2020'
  Publish Date: '2021:01:01:13:11:36'
- imdbID: aa0000004
  Original Title: Schiller - Atemlos
  Polish Title: Schiller - Atemlos Live
  imdbLink: http://www.imdb.com/title/aa0000004/
  Rating: N/A
  Genre: Music
  Release Year: '2010'
  Publish Date: '2017:07:02:09:24:46'
- imdbID: aa0000006
  Original Title: Schiller - Future
  Polish Title: Schiller - Future Live
  imdbLink: http://www.imdb.com/title/aa0000006/
  Rating: N/A
  Genre: Music
  Release Year: N/A
  Publish Date: '2017:07:02:12:54:56'
- imdbID: tt3812422
  Original Title: Schiller - Live in Berlin
  Polish Title: Schiller - Live in Berlin
  imdbLink: http://www.imdb.com/title/tt3812422/
  Rating: N/A
  Genre: Documentary
  Release Year: '2009'
  Publish Date: '2017:01:29:11:12:40'
- imdbID: aa0000011
  Original Title: Schiller - Live in Tehran
  Polish Title: Schiller - Live in Tehran
  imdbLink: http://www.imdb.com/title/aa0000011/
  Rating: N/A
  Genre: Music
  Release Year: N/A
  Publish Date: '2020:04:19:10:00:38'
- imdbID: aa0000008
  Original Title: Schiller - Sehnsucht
  Polish Title: Schiller - Sehnsucht Live
  imdbLink: http://www.imdb.com/title/aa0000008/
  Rating: N/A
  Genre: Music
  Release Year: N/A
  Publish Date: '2017:07:02:13:42:40'
- imdbID: aa0000010
  Original Title: Schiller - Sonne
  Polish Title: Schiller - Sonne Live
  imdbLink: http://www.imdb.com/title/aa0000010/
  Rating: N/A
  Genre: Music
  Release Year: N/A
  Publish Date: '2017:07:01:17:35:40'
- imdbID: aa0000009
  Original Title: Schiller - Symphonia
  Polish Title: Schiller - Symphonia Live
  imdbLink: http://www.imdb.com/title/aa0000009/
  Rating: N/A
  Genre: Music
  Release Year: N/A
  Publish Date: '2020:04:20:00:04:20'
- imdbID: tt3812422
  Original Title: Schiller - Tagtraum
  Polish Title: Schiller - Tagtraum
  imdbLink: http://www.imdb.com/title/tt3812422/
  Rating: N/A
  Genre: Documentary
  Release Year: '2009'
  Publish Date: '2021:03:21:12:05:37'
- imdbID: aa0000007
  Original Title: Schiller - Zeitreise
  Polish Title: Schiller - Zeitreise Live
  imdbLink: http://www.imdb.com/title/aa0000007/
  Rating: N/A
  Genre: Music
  Release Year: N/A
  Publish Date: '2018:05:31:18:28:29'
- imdbID: aa0000002
  Original Title: THePETEBOX
  Polish Title: THePETEBOX
  imdbLink: http://www.imdb.com/title/aa0000002/
  Rating: N/A
  Genre: Music
  Release Year: '2013'
  Publish Date: '2015:02:13:20:09:00'
- imdbID: tt0110758
  Original Title: Pink Floyd - Pulse
  Polish Title: Pink Floyd - Pulse
  imdbLink: http://www.imdb.com/title/tt0110758/
  Rating: '9.3'
  Genre: Documentary, Music
  Release Year: '1994'
  Publish Date: '2023:10:31:00:50:18'
- imdbID: tt2414166
  Original Title: Led Zeppelin - Celebration Day
  Polish Title: Led Zeppelin Celebration Day
  imdbLink: http://www.imdb.com/title/tt2414166/
  Rating: '9.0'
  Genre: Documentary, Music
  Release Year: '2012'
  Publish Date: '2012:12:10:23:40:32'
- imdbID: tt1132456
  Original Title: David Gilmour - Remember That Night
  Polish Title: David Gilmour - Remember That Night
  imdbLink: http://www.imdb.com/title/tt1132456/
  Rating: '8.9'
  Genre: Documentary, Music
  Release Year: '2007'
  Publish Date: '2016:08:08:20:53:24'
- imdbID: tt0271061
  Original Title: Metallica - S&M
  Polish Title: Metallica S&M
  imdbLink: http://www.imdb.com/title/tt0271061/
  Rating: '8.8'
  Genre: Documentary, Music
  Release Year: '2000'
  Publish Date: '2020:04:19:14:03:15'
- imdbID: tt7359702
  Original Title: David Gilmour - Live At Pompeii
  Polish Title: David Gilmour - Na żywo w Pompei
  imdbLink: http://www.imdb.com/title/tt7359702/
  Rating: '8.7'
  Genre: Documentary, Music
  Release Year: '2017'
  Publish Date: '2018:09:01:23:30:40'
- imdbID: tt0069090
  Original Title: Pink Floyd - Live at Pompeii
  Polish Title: Pink Floyd - Live at Pompeii
  imdbLink: http://www.imdb.com/title/tt0069090/
  Rating: '8.7'
  Genre: Documentary, Music
  Release Year: '1972'
  Publish Date: '2020:02:22:14:40:08'
- imdbID: tt2122272
  Original Title: Adele - Live At The Royal Albert Hall
  Polish Title: Adele - Na żywo w Royal Albert Hall
  imdbLink: http://www.imdb.com/title/tt2122272/
  Rating: '8.6'
  Genre: Documentary, Music
  Release Year: '2011'
  Publish Date: '2018:05:31:10:48:37'
- imdbID: tt1388331
  Original Title: Chris Botti - in Boston
  Polish Title: Chris Botti - in Boston
  imdbLink: http://www.imdb.com/title/tt1388331/
  Rating: '8.6'
  Genre: Music
  Release Year: '2009'
  Publish Date: '2018:05:05:17:38:13'
- imdbID: tt0244296
  Original Title: Roger Waters - The Wall Live in Berlin
  Polish Title: Roger Waters - The Wall na żywo w Berlinie
  imdbLink: http://www.imdb.com/title/tt0244296/
  Rating: '8.6'
  Genre: Documentary, Music
  Release Year: '1990'
  Publish Date: '2020:11:06:13:30:52'
- imdbID: tt2122472
  Original Title: Shakira - Live From Paris
  Polish Title: Shakira - Live From Paris
  imdbLink: http://www.imdb.com/title/tt2122472/
  Rating: '8.5'
  Genre: Documentary, Music
  Release Year: '2011'
  Publish Date: '2020:04:20:10:55:30'
- imdbID: tt0097181
  Original Title: Pink Floyd - Delicate Sound Of Thunder
  Polish Title: Pink Floyd - Delicate Sound Of Thunder
  imdbLink: http://www.imdb.com/title/tt0097181/
  Rating: '8.4'
  Genre: Music
  Release Year: '1989'
  Publish Date: '2022:05:14:09:12:05'
- imdbID: tt28814949
  Original Title: Taylor Swift - The Eras Tour
  Polish Title: Taylor Swift - The Eras Tour
  imdbLink: http://www.imdb.com/title/tt28814949/
  Rating: '8.3'
  Genre: Documentary, Music
  Release Year: '2023'
  Publish Date: '2023:12:13:21:31:12'
- imdbID: tt0084503
  Original Title: Pink Floyd - The Wall
  Polish Title: Ściana
  imdbLink: http://www.imdb.com/title/tt0084503/
  Rating: '8.0'
  Genre: Animation, Drama, Fantasy, Music, Musical
  Release Year: '1982'
  Publish Date: '2018:11:19:07:57:47'
- imdbID: tt0214952
  Original Title: Mike Oldfield - Tubular Bells III
  Polish Title: Mike Oldfield - Tubular Bells III
  imdbLink: http://www.imdb.com/title/tt0214952/
  Rating: '7.8'
  Genre: Documentary, Music
  Release Year: '1999'
  Publish Date: '2020:04:13:18:05:49'
- imdbID: tt0075617
  Original Title: ABBA The Movie
  Polish Title: ABBA. Film
  imdbLink: http://www.imdb.com/title/tt0075617/
  Rating: '6.4'
  Genre: Documentary, Comedy, Drama
  Release Year: '1977'
  Publish Date: '2023:07:23:15:44:05'
- imdbID: tt0978039
  Original Title: Loreena McKennitt -  Nights from the Alhambra
  Polish Title: Loreena McKennitt -  Nights from the Alhambra
  imdbLink: http://www.imdb.com/title/tt0978039/
  Rating: '6.4'
  Genre: Music
  Release Year: '2007'
  Publish Date: '2014:04:04:12:50:04'

Now, the problem is, that I can sort by all of the attributes defined in input_select.sort_by, with exception of Release Date. Not only applying this atrribute do not properly order values, but also sort order has no influence on sorting. I made some experiment in Developers tools, changing this parameters manually in template and I can see that some unpredictable sort order is applied only:


Trying to get it resolved, I tried to use your dictionary method to create the sensor, and apply int filter to 1Release Year1, to perhaps made sorting possible:

      - name: movies_fltr
        state: 'on'
        attributes:
          details: > 
            {% set attr = states('input_select.sort_by') %}
            {% set ord = is_state('input_boolean.sort_order', 'on') %}
            {% set lib = states('input_select.library') %}
            {% set movies_all = state_attr('sensor.movies_list', 'list') | sort(attribute=attr, reverse=ord) | selectattr('Library', 'in', lib) %}
            {% set movies_fltr = namespace(movies=[]) %}
            {% for m in movies_all %}
              {% set movies_fltr.movies = movies_fltr.movies + [dict(
                imdbID=m.imdbID,
                imdbLink=m.imdbLink,
                Rating=m.Rating,
                Genre=m.Genre,
                Original_Title=m['Original Title'],
                Polish_Title=m['Polish Title'],
                Release_Year=m.ReleaseYear|int(2000),
                Publish_Date=m['Publish Date']
                )] %}
            {% endfor %}
            {{ movies_fltr.movies }}

It creates almost identical output, the only difference is that Release Year is now integer, not a string. But it did not helped to resolve issue with sorting, still the same problem. Additionally issues I mentioned earlier:

  • is it possible using this method to create json keys with spaces in names? As I use these for filtering in UI it just looks better to have Original Title vs. OriginalTitle or Original-Title or Original_Title.
  • I found no way to apply ‘paging’… Using loop, as in my example I can easily filter objects into groups of 100 items for template output. This is quite serious issue, as in my library I have >6k movies and each of thes takes ~250 characters, leading to total size of unfiltered attribute being +1.5M characters. I know, I should probably use some database for this rather than json file, but this is what I already have and maintain for years :slight_smile:

Any suggestions to this filtering issue? probably something very simple and stupid, but I can’t find what’s wrong…

I suggest you try sorting movies_fltr.movies instead of movies_all.

I don’t believe the dict() function allows you to do that.

The first 100 items in movies_fltr.movies.

{{ movies_fltr.movies[:100] }}

The next 100 items in movies_fltr.movies.

{{ movies_fltr.movies[101:200] }}

Can you provide me with the output of this template? It will help me understand what kind of data you’re working with.

{{ state_attr('sensor.movies_list', 'list')[:10] }}

Here is the output of tyour templaye:

[
  {
    "Publish Date": "2011:10:27:05:40:06",
    "Library": "3D Movies",
    "Original Title": "Avatar 3D",
    "Polish Title": "Avatar",
    "imdbLink": "http://www.imdb.com/title/tt0499549/",
    "imdbID": "tt0499549",
    "Genre": "Action, Adventure, Fantasy",
    "Rating": "7.9",
    "ReleaseYear": "2009"
  },
  {
    "Publish Date": "2016:12:14:05:29:52",
    "Library": "3D Movies",
    "Original Title": "Beowulf 3D",
    "Polish Title": "Beowulf",
    "imdbLink": "http://www.imdb.com/title/tt0442933/",
    "imdbID": "tt0442933",
    "Genre": "Animation, Action, Adventure",
    "Rating": "6.3",
    "ReleaseYear": "2007"
  },
  {
    "Publish Date": "2011:11:03:18:48:34",
    "Library": "3D Movies",
    "Original Title": "Clash Of The Titans 3D",
    "Polish Title": "Starcie Tytanów",
    "imdbLink": "http://www.imdb.com/title/tt0800320/",
    "imdbID": "tt0800320",
    "Genre": "Action, Adventure, Fantasy",
    "Rating": "5.8",
    "ReleaseYear": "2010"
  },
  {
    "Publish Date": "2014:03:08:17:17:32",
    "Library": "3D Movies",
    "Original Title": "Gravity 3D",
    "Polish Title": "Grawitacja",
    "imdbLink": "http://www.imdb.com/title/tt1454468/",
    "imdbID": "tt1454468",
    "Genre": "Sci-Fi, Thriller",
    "Rating": "8.0",
    "ReleaseYear": "2013"
  },
  {
    "Publish Date": "2011:10:31:13:38:12",
    "Library": "3D Movies",
    "Original Title": "Green Lantern 3D",
    "Polish Title": "Zielona Latarnia",
    "imdbLink": "http://www.imdb.com/title/tt1133985/",
    "imdbID": "tt1133985",
    "Genre": "Action, Adventure, Sci-Fi",
    "Rating": "5.7",
    "ReleaseYear": "2011"
  },
  {
    "Publish Date": "2013:05:22:15:23:18",
    "Library": "3D Movies",
    "Original Title": "I Robot 3D",
    "Polish Title": "Ja, Robot",
    "imdbLink": "http://www.imdb.com/title/tt0343818/",
    "imdbID": "tt0343818",
    "Genre": "Action, Mystery, Sci-Fi",
    "Rating": "7.1",
    "ReleaseYear": "2004"
  },
  {
    "Publish Date": "2016:08:03:22:42:00",
    "Library": "3D Movies",
    "Original Title": "Inside Out 3D",
    "Polish Title": "W głowie się nie mieści",
    "imdbLink": "http://www.imdb.com/title/tt2096673/",
    "imdbID": "tt2096673",
    "Genre": "Animation, Adventure, Comedy",
    "Rating": "8.4",
    "ReleaseYear": "2015"
  },
  {
    "Publish Date": "2012:06:10:10:08:10",
    "Library": "3D Movies",
    "Original Title": "John Carter 3D",
    "Polish Title": "John Carter",
    "imdbLink": "http://www.imdb.com/title/tt0401729/",
    "imdbID": "tt0401729",
    "Genre": "Action, Adventure, Fantasy",
    "Rating": "6.6",
    "ReleaseYear": "2012"
  },
  {
    "Publish Date": "2013:05:22:21:02:24",
    "Library": "3D Movies",
    "Original Title": "Jurassic Park 3D",
    "Polish Title": "Park Jurajski",
    "imdbLink": "http://www.imdb.com/title/tt0107290/",
    "imdbID": "tt0107290",
    "Genre": "Adventure, Sci-Fi, Thriller",
    "Rating": "8.0",
    "ReleaseYear": "1993"
  },
  {
    "Publish Date": "2011:11:02:23:24:00",
    "Library": "3D Movies",
    "Original Title": "Priest 3D",
    "Polish Title": "Ksiądz",
    "imdbLink": "http://www.imdb.com/title/tt0822847/",
    "imdbID": "tt0822847",
    "Genre": "Action, Fantasy, Horror",
    "Rating": "5.7",
    "ReleaseYear": "2011"
  }
]

I’ll test your other suggestions as soon as I et soime free time!

As an experiment, let me know if this produces a list of fifteen 3D Movies sorted by Release Year. (It works with the sample data you provided.)

{{ (state_attr('sensor.movies_list', 'list')
  | selectattr('Library', 'eq', '3D Movies')
  | sort(attribute='ReleaseYear', reverse=false)
  | list)[:15] }}

NOTE

I think the reason your template failed to sort correctly is because in the source data the key’s name is ReleaseYear but the string you used is from your Input Select which is Release Year.

The sort function couldn’t find a Release Year key in the dictionary so it left the data unsorted.

Yes, it works! The template you provided gives proper output and I can change sort order.

This is it !
I was so sure that during convertion for original source json file that has the key ReleaseYear I changed this key to more readible form Release Year that I went blind…
Thanks so much for helping with this (as expected very simple) issue, thatkI overlooked in hours of searching!
Now I’ll probably start from fixing my original scanning script to get proper key at first and then I’ll incorporate your trick with [:100] to get to proper dictionary solution!
BTW, how to make the post double solution :slight_smile: :+1:

1 Like