Without knowing what version you are currently running I can only guess that you are currently running an 0.116.x release which had a well documented issue with casting to a nest hub. It was patched with one of the subsequent minor releases.
Then I would personally first check the stream from the doorbird in vlc to ensure the source isn’t the issue.
If streaming worked prior to v116 you may need to raise the issue using the GitHub template. I believe there were some significant changes to streaming in 116 to add audio (hence the casting issues in early 116 releases). I cast Unifi Protect cameras to my nest hub in 117 without issue.
Might be worth checking if this is the issue you have.