Notify.html5 actions only work when site open

I am trying to run notification actions on an android phone using the notify.html5 platform. Pressing the action button only works when the web app is open/active on the screen, then it runs the automation but if the web app isn’t active it doesn’t seem to do anything. Has anyone run into this?

{
    "message": "Test Message",
    "data": {
        "entity_id": "alert.garage_door_alert",
        "service": "alert.turn_off",
        "actions": [{
            "action": "run_service",
            "type": "clicked",
            "title": "turn off"
        }]
    }
}