With a bit of googling I found a Twitter API that returns a json file with the follower count for a user. It seems Twitter use it for widget buttons so I repurposed it as a HA Rest Sensor. Hopefully Twitter won’t remove it any time soon.
Just add the sensor to your configuration.yaml and replace the username at the end of the URL with your own to get your follower count.
sensor:
- platform: rest
resource: https://cdn.syndication.twimg.com/widgets/followbutton/info.json?screen_names=AndrewCodeChimp
name: Twitter Followers
value_template: '{{ value_json[0].followers_count }}'
scan_interval: 43200 # Every 12 hours in seconds