Check out @blakeblackshear project he has been working on.
This uses a docker container running tensorflow instead of the HA component.
He has done a lot of work and it sounds like it might be what you are looking for - at least until HA has native support for something like Coral.