Turns out this was completely my bad…
I am using my ISPs router in Bridge mode (cannot be circumvented) and then using Google WiFi for my actual network.
I forwarded the ports on my Google WiFi, but not on my ISPs router. Turns out both is needed and this immediately fixes the issue.