So from what I understand having two routers and two networks isn´t a good idea but I almost got it to work 
In the image below is map of my current network. Everthing works, I can control all my entities through my phone even tho its not connected to the same network but I just can’t get my desktop to talk to my HA Raspberry pi. Any suggestions?
Probably because you are going through the cloud?
Yes, but I can also use the philips hue app, smartthings app, roborock app to control the entitites. Not sure if all those are cloud based aswell
Disconnect your phone from WiFi. Can you still control devices? Yes? then they are in the cloud.
But, your network map looks unnecessarily complex. Why do you have three networks?
Yeah, they were all cloud.
I have “only” two networks. The first router is a shared that only splits the incoming fiber connection between my appartment and my neighbour. I will probably scrap this idea and might just go and setup a guest network for my IoT instead.
Would certainly simplify things.
