After some poking around, I figured it out. Putting it here in case someone else has same issue. In the Settings → Add-Ons, go to Matter Server → Open WebUI. Under Nodes, the removed node will be listed as failed (or something like that). Click on that node and select Remove. This will remove the device and entities.