Ah gotchu.
you can use state_attr != none for that.
but it will also trigger as false if the light is off…
i havent found a way to check if an entity has an attribute sometimes. because that attribute dissapears when the light is off i cant really test for it.
{{ state_attr('light.office_main_light', 'rgb_color') != none }}
{{ state_attr('light.kitchen_cabinet_light', 'rgb_color') != none }}
{{ state_attr('light.kitchen_down_light_1', 'rgb_color') != none }}
so best you can do is test for if a light is on and it has the rgb_color attribute. you can do this in 2 ways. either a combined statement like this. where you then only have 2 choices.
{% if states('light.office_main_light') == 'on' and state_attr('light.office_main_light', 'rgb_color') != none %}
Change Background to Light Color
{% else %}
Light is either Off or Doesnt Support rgb_color Attribute.
{% endif %}
or what i think is better in this instance, a nested if statement that then gives you 3 options like this.
{% if states('light.kitchen_down_light_1') == 'on' %}
{%- if state_attr('light.kitchen_down_light_1', 'rgb_color') != none -%}
Change Background to Light Color
{%- else -%}
Set a Standard Background Color Because Light Doesnt Support rgb_color Attribute
{%- endif -%}
{% else %}
Light is Off
{% endif %}
Edit: can also do it with an elif instead to have all 3 options. Probably a bit easier to read too 
{% if states('light.kitchen_down_light_1') == 'on' and state_attr('light.kitchen_down_light_1', 'rgb_color') != none %}
Change Background to Light Color
{% elif states('light.kitchen_down_light_1') == 'on' %}
Set a Standard Background Color Because Light Doesnt Support rgb_color Attribute
{% else %}
Light is Off
{% endif %}
