After knowing thats its not only the CPU but also the memory; I started to search for memory leaks. And I probably found my issue, it has to do with the Synology component and the HTTPS connection to the Synology backend, I switched is to HTTP and my issue seems to be gone.
Related topic:
And related Github issue from 2017