For anyone who wants to do the same, here’s my Lovelace card with card-mod to hide the Code input and keyboard.
type: alarm-panel
states:
- arm_home
- arm_away
entity: alarm_control_panel.home_alarm_panel
name: Home Alarm
style: |
#keypad, #alarmCode {
display: none !important;
}