Migrate HTTP stack to WSGI-based framework

Right now we are running my homegrown web framework on top of the Python built-in HTTP server. It works ok for the most time but it is not battle tested and I doubt all standards are followed correctly.

So it’s time to migrate.

We’ve talked about it in the chat and most people (including me) are leaning towards using a solution that implements WSGI, the Python web framework standard. We are thinking about using either Werkzeug or Flask (which builds on top of Werkzeug).

Besides a web framework we will also need a wsgi server.

The requirements that we currently have for a web framework:

  • url routing
  • serve a dynamic page to serve the frontend
  • serve static assets
  • serve our REST API
  • Serve our eventstream (keep connection open and write to it) (we can change SSE implementation/probably should for better browser support)
  • Authentication
  • Sessions

The requirements that we currently have for a WSGI server:

  • HTTP 1.1
  • Can handle multiple simultaneous requests
  • Support for SSL
  • Support for GZIP compression of responses
  • Pure Python (becauses otherwise a pain to install)

Other good to know things:

  • Home Assistant already includes jinja2 for templating.
3 Likes

Great. I took a quick look at this earlier and I don’t think it will be too painful. More coherent thoughts from me in the morning…

Perhaps it’s worth to have a quick look at aiohttp, too.

I can’t speak to the level of activity on these today but, in the past working with flask I’ve found these extensions useful. They saved me a lot of time in the end.

httpauth
https://flask-httpauth.readthedocs.org/en/latest/

flask-RESTful
http://flask-restful-cn.readthedocs.org/en/0.3.4/

Changing to WSGI will allow me to finally implement python-socket.io as an additional event transport (and our first built-in bidirectional one, to supplement the currently implemented unidirectional server-sent events/EventStream).

2 Likes

Just to give folks an update… I’m about 80% done with a draft of this (based on Werkzeug, using Eventlet as a WSGI server, and static3 to handle static file serving). Once it’s done, I’ll stick it in a branch for discussion (I’m not shooting for polished here, just ‘rough draft’). The bulk of the remaining work to hit ‘draft’ status is auth/sessions and converting some more API views.

I’m developing this as a new component, called ‘wsgi’. Eventually we’ll want to rename it ‘http’ so that existing configs work, but I think we should ship them in parallel for a release or two (announcing from the start that the ‘wsgi’ component name is temporary, and will need to be changed to ‘http’ at some point (and no backwards compatible mapping will be added, as the wsgi name was only intended for the experimental version)). This will allow folks who don’t feel comfortable running a dev version to still help with testing.

My first draft is going to break the API event stream. We are going to need to migrate to a ‘real’ event protocol (pure websockets, socket.io, whatever), but that will require frontend work as well, and Polymer is a little outside my wheelhouse at the moment.

I’d like to get the experimental version in .19. The scope for the initial version is simply ‘equivalence’ with the current http component. It needs to do everything the http component does, exactly as it does it (quirks included), and shouldn’t worry about doing anything more. The goal is simply to set a foundation that will make it easy to add more functionality (better IPv6 support, alternate webroots, etc) in future releases. Those things may happen before the wsgi stack becomes the default, or they may not, I’m not overly concerned with that.

2 Likes

So as it turns out, having newborn twins takes up a ton of your time… I haven’t made a ton of progress on this since my last post. To that end, here’s my work-in-progress so far: https://github.com/JshWright/home-assistant/commit/7848c0cde699b701b8f98b6618e341af8cffd020

1 Like