Increase of devices every time I touch Google Assistant integration

So I am pulling hairs in frustration right now. I have my HassIO setup with my Google Home/Assistant to control lights etc. Now, I have about 5 devices, some lights, some switches, and one group, where some of the lights are in. However my Google Home Mini thinks that I have 31 devices, and will tell me that some devices are unavailable etc. "Hey google, sync devices does nothing. I have tried excluding everything with:

cloud:
  google_actions:
    filter:
      exclude_domains:
        - switch
        - light
        - climate
        - group

And all devices are gone in my Google assistant app on my phone, but the Google Home can still control them, and tell me that it is turning on 31 devices every time I say “Turn on everything”.

How can I solve this, without starting all over with my config again?

EDIT: So this seems to be the problem I’m having. I just have no idea why I am able to control my Sonoff switches when they are excluded in my config.