Can I show icons inside templates

You can template in Mushroom cards. Just use the type: custom:mushroom-template-card

type: custom:mushroom-template-card
entity: light.pc_lights
icon: |
   {{ 'mdi:lightbulb-off' if states(config.entity) == 'off' else 'mdi:lightbulb'}}
icon_color: |-
    {{ 'grey' if states(config.entity) == 'off' else 'yellow'}}
primary: Light Example

For Mushroom related questions, check out the main threads here and here