Getting played album info from OpenAI

I’d like to share some work I did to get descriptive infomration about album being played by media_player to somehow enrich listening experience. The goal was to create dashboard a bit resembling Roon web display.
Here is the outcome so far (not yet perfect, need to be finetuned to screen size, I hope to finally have something more responsive to screen size and orientation):

Here is the code I used.
First automation to retrieve album information from ChatGPT, using native service responce functionality of HA 2023.7 (thanks a lot @Taras and @petro to resolve some details this code!):

automation:
  - id: 'update_media_info'
    alias: Update Media Info
    initial_state: true
    variables:
      player: >-
        {{ trigger.entity_id }}
      artist: >-
        {% set string=trigger.entity_id %}
        {% set result=state_attr(string, 'media_artist') %}
        {{ result }}
      album: >-
        {% set string=trigger.entity_id %}
        {% set result=state_attr(string, 'media_album_name') %}
        {{ result }}
      output: >-
        {% set string=trigger.entity_id %}
        {% set string=string.split('.') %}
        {% set result="input_text." + string[1] + "_media_info" %}
        {{ result }}
    trigger:
    - platform: state
      entity_id:
        - media_player.audiocast
        - media_player.denon_heos_s750h
        - media_player.marantz_sacd30n
        - media_player.volumio_2
      attribute: media_album_name
    condition: []
    action:
      - service: conversation.process
        data:
          agent_id: 9194fb3fee4a1e2220e2f47a524fce92
          text: Tell me about {{ artist }}'s album {{ album }} in less than 150 words
        response_variable: chatgpt
      - event: chatgpt_response
        event_data:
          entity_id: "{{ trigger.entity_id }}"
          response: "{{chatgpt.response.speech.plain.speech | trim | replace('\"','')}}"

Please noet that this automation is created to support multiple media players. You need to list all of media_player you want to track in trigger section, under entity_id. In this iteration automation is triggered every time media_album_name attribute of any of these player changes and retrieves desired information from OpenAI. At the moment it is configured to retrieve response no longer that 150 words, that is reasonable text size for my use (to fit on the available screen space).
Once description is received from OpenAI event is triggered to update template sensor corresponding to specic media_player. So for each one you need to create separate sensor, following following template:

template:
  - trigger:
      - platform: event
        event_type: chatgpt_response
        event_data:
          entity_id: media_player.volumio_2
      - platform: state
        entity_id:
          - media_player.volumio_2
        attribute: media_title
    sensor:
      - name: volumio_2_album_description
        state: 'on'
        attributes:
          album_description: >
            {% if trigger.platform == 'event' %}
              {{ trigger.event.data.response }}
            {% else %}
              {{ this.attributes.album_description | default('') }}
            {% endif %}   
          album_title: "{{ state_attr('media_player.volumio_2', 'media_album_name') }}"
          artist_name: "{{ state_attr('media_player.volumio_2', 'media_artist') }}"
          song_title: "{{ state_attr('media_player.volumio_2', 'media_title') }}"
          album_art: "{{ state_attr('media_player.volumio_2', 'entity_picture') }}"

Please note that in this iimplementation sensor state is permanently set to ‘on’ and all additional media information is stored as attributes. For easier templating of the card sensor will duplicate infomrmation about artist, album, track and album art, so it has douple triggers; event generated by previous automation will update album_description attribute, played track change will additionally update other attributes.
Obviously you can retrieve these attributes directly from media_played, so this template sensor can be simplified.
Finally here is the dashboard code:

type: custom:config-template-card
variables:
  - states['input_select.media_players'].state
  - states[vars[0]].attributes.friendly_name
  - >-
    states[vars[0].replace('media_player', 'sensor') +
    '_album_description'].attributes.song_title
  - >-
    states[vars[0].replace('media_player', 'sensor') +
    '_album_description'].attributes.artist_name
  - >-
    states[vars[0].replace('media_player', 'sensor') +
    '_album_description'].attributes.album_title
  - >-
    states[vars[0].replace('media_player', 'sensor') +
    '_album_description'].attributes.album_description
  - >-
    states[vars[0].replace('media_player', 'sensor') +
    '_album_description'].attributes.album_title
  - >-
    states[vars[0].replace('media_player', 'sensor') +
    '_album_description'].attributes.album_art
entities:
  - ${vars[0]}
card:
  type: picture-elements
  image: ${vars[7]}
  elements:
    - type: image
      image: ${vars[7]}
      style:
        top: 49%
        left: 49%
        width: 102%
    - type: image
      image: /local/overlay.png
      style:
        top: 49%
        left: 49%
        width: 120%
    - type: image
      image: /local/overlay2.png
      style:
        top: 49%
        left: 49%
        width: 120%
    - type: custom:layout-card
      style:
        top: 0%
        left: 0%
        transform-origin: left top
        transform: translate(-0%,-0%)
      layout_type: grid
      layout_options:
        grid-template-columns: 3% 40% 1% auto 4%
        grid-template-rows: 2% auto 200px
        grid-template-areas: |
          "a a a a a"
          "b art c desc d"
          "f info info info g"
        mediaquery:
          '(max-width: 700px)':
            grid-template-columns: 100%
            grid-template-rows: auto auto auto
            grid-template-areas: |
              "art"
              "desc"
              "info"
      cards:
        - entity: ${vars[0]}
          type: custom:mini-media-player
          artwork: full-cover-fit
          scale: '1.5'
          hide:
            mute: true
            icon: true
            power: true
            volume: true
            controls: true
            source: true
            name: true
            info: true
            progress: true
          view_layout:
            grid-area: art
        - type: custom:hui-element
          view_layout:
            grid-area: desc
          card_type: custom:vertical-stack-in-card
          cards:
            - type: markdown
              content: ${vars[5]}
              card_mod:
                style: |
                  ha-card {
                    background: transparent;
                    font-size: 0.8em;
                    line-height: 1.2
                  }
          card_mod:
            style: |
              ha-card {
                background: transparent;
                font-size: 1.5em;
                line-height: 1.2
              }
        - type: custom:vertical-stack-in-card
          card_mod:
            style: |
              ha-card {
                background: transparent
              }
          horizontal: false
          view_layout:
            grid-area: info
          cards:
            - entity: ${vars[0]}
              type: custom:mini-media-player
              info: scroll
              artwork: none
              scale: '1.8'
              hide:
                icon: true
                power: true
                volume: true
                name: true
                source: true
                shuffle: true
                repeat: true
                info: true
                progress: false
                runtime: false
                runtime_remaining: false
                mute: true
                controls: true
            - type: custom:vertical-stack-in-card
              card_mod:
                style: |
                  ha-card {
                    background: transparent
                  }
              horizontal: true
              cards:
                - type: markdown
                  card_mod:
                    style: |
                      ha-card {
                        background: transparent;
                        line-height: 1.1
                        }
                  content: >-
                    ${"<font color=white><font size=6.0em>" + vars[2] + "<font
                    color=grey></br>" + "<font size=5.5em>" + vars[3] +
                    "<font></br>" + "<font size=5.0em>" + vars[4] + "<font>"}
                - type: custom:vertical-stack-in-card
                  card_mod:
                    style: |
                      ha-card {
                        background: transparent
                      }
                  horizontal: false
                  cards:
                    - type: custom:vertical-stack-in-card
                      card_mod:
                        style: |
                          ha-card {
                            background: transparent
                          }
                      horizontal: true
                      cards:
                        - type: custom:button-card
                          entity: media_player.audiocast
                          color_type: card
                          triggers_update: input_select.media_players
                          styles:
                            card:
                              - background-color: |
                                  [[[
                                    if (states['input_select.media_players'].state == 'media_player.audiocast') return 'rgba(0,0,0,0.5)';
                                    else return 'rgba(0,0,0,0.2)';
                                  ]]]
                            name:
                              - color: |
                                  [[[
                                    if (states['input_select.media_players'].state == 'media_player.audiocast') return 'rgb(255,255,255)';
                                    else return 'rgba(128,128,128)';
                                  ]]]
                          aspect_rato: 4/1
                          show_icon: false
                          name: Audiocast
                          tap_action:
                            action: call-service
                            service: input_select.select_option
                            service_data:
                              entity_id: input_select.media_players
                              option: media_player.audiocast
                        - type: custom:button-card
                          entity: media_player.denon_heos_s750h
                          color_type: card
                          triggers_update: input_select.media_players
                          styles:
                            card:
                              - background-color: |
                                  [[[
                                    if (states['input_select.media_players'].state == 'media_player.denon_heos_s750h') return 'rgba(0,0,0,0.5)';
                                    else return 'rgba(0,0,0,0.2)';
                                  ]]]
                            name:
                              - color: |
                                  [[[
                                    if (states['input_select.media_players'].state == 'media_player.denon_heos_s750h') return 'rgb(255,255,255)';
                                    else return 'rgba(128,128,128)';
                                  ]]]
                          aspect_rato: 4/1
                          show_icon: false
                          name: Denon
                          tap_action:
                            action: call-service
                            service: input_select.select_option
                            service_data:
                              entity_id: input_select.media_players
                              option: media_player.denon_heos_s750h
                        - type: custom:button-card
                          entity: media_player.marantz_sacd30n
                          color_type: card
                          triggers_update: input_select.media_players
                          styles:
                            card:
                              - background-color: |
                                  [[[
                                    if (states['input_select.media_players'].state == 'media_player.marantz_sacd30n') return 'rgba(0,0,0,0.5)';
                                    else return 'rgba(0,0,0,0.2)';
                                  ]]]
                            name:
                              - color: |
                                  [[[
                                    if (states['input_select.media_players'].state == 'media_player.marantz_sacd30n') return 'rgb(255,255,255)';
                                    else return 'rgba(128,128,128)';
                                  ]]]
                          aspect_rato: 4/1
                          show_icon: false
                          name: Marantz
                          tap_action:
                            action: call-service
                            service: input_select.select_option
                            service_data:
                              entity_id: input_select.media_players
                              option: media_player.marantz_sacd30n
                        - type: custom:button-card
                          entity: media_player.volumio_2
                          color_type: card
                          triggers_update: input_select.media_players
                          styles:
                            card:
                              - background-color: |
                                  [[[
                                    if (states['input_select.media_players'].state == 'media_player.volumio_2') return 'rgba(0,0,0,0.5)';
                                    else return 'rgba(0,0,0,0.2)';
                                  ]]]
                            name:
                              - color: |
                                  [[[
                                    if (states['input_select.media_players'].state == 'media_player.volumio_2') return 'rgb(255,255,255)';
                                    else return 'rgba(128,128,128)';
                                  ]]]
                          aspect_rato: 4/1
                          show_icon: false
                          name: Volumio
                          tap_action:
                            action: call-service
                            service: input_select.select_option
                            service_data:
                              entity_id: input_select.media_players
                              option: media_player.volumio_2
                    - entity: ${vars[0]}
                      type: custom:mini-media-player
                      info: scroll
                      artwork: none
                      scale: '1.4'
                      hide:
                        icon: true
                        power: false
                        volume: true
                        name: true
                        source: true
                        shuffle: false
                        repeat: false
                        info: true
                        progress: true
                        runtime: true
                        runtime_remaining: true
                        mute: false
                        controls: false

To work this dashboard requires 2 files (overlay.png and overlay2.png that are just black rectangles with various transparency) to darken background. As you might notice there are additional buttons incorporated into this dashboard, that corresponds to media_players I use. In order to use these you also need to define input_select helper that allows to select one to be used in dashboard:

input_select:
  media_players:
    name: List of Media Players
    options:
      - media_player.audiocast
      - media_player.denon_heos_s750h
      - media_player.marantz_sacd30n
      - media_player.volumio_2

This design (visualisation) is not perfect. First it does not change automatically with change of selected media_player. It is also subject of frequent redraws when some media_players are used (I found it to be relatd to displaying progress. Some media_players report progress every second, some every 10 secodns and some only at the beginning of play and at stop. In this last case progress bar size is calculated by media player card based on start time and elapsed time, but is refreshed every 30 seconds, causing progress bar to restart (instead of showing actual progress) and redraw entire dashbord. If this is the case and it annoys you, it is always possible to remove displaying progress bar from configuration.

10 Likes

Thanks for the great post.

Automation stopped due to below error:

Stopped because an error was encountered at 13 July 2023 at 17:48:34 (runtime: 0.03 seconds)

invalid agent ID for dictionary value @ data['agent_id']

I’m a bit confused here, is the agent_id the OpenAI api key?
I figured out it is not the api key, so the question where can I find the agent_id?

Go to developer tools, services and choose OpenAI conversation:


Then switch to YAML mode and you will see the agent_id for your instance:

This looks very interesting. Wondering … would this cover an entire screen of a tablet? Could it also be displayed on a TV?
Is there any prerequisite to running this? I would assume OpenAI, Is it just as easy as installing the Integration? Anything else needed?

Well, I guess this would require at least browser mod to be installed, to be able to hide top menu and sidebar (screenshot provides sort of crop from entire screen). Unless you’d plan for using display also as sort of ‘magic mirror’, then having menus might be usefull.
On top of OpenAI I also use config template card, layout card, mini media player, vertyical stack in card (for dashboard itself) - but this is probably something than can be tailored to personal preferences of components to be used for UI. For getting info just OpenAI integration is required.

Thanks a lot → found :ok_hand:

  response: >-
    Sorry, I had a problem talking to OpenAI: You exceeded your current quota,
    please check your plan and billing details.

Unfortunately :pensive:

Yeap, I hit this problem too while testing (though do not recal exact error I’ve seen). This led me to ordering paid subscription - over 3 months of trial I only used ~65% of tokens, so with current pricing model I’d expect this to cost me ~$1/month… still can afford :slight_smile: