Hi Hogster and thanks for the input!
TL/DR: I really do waffle on here, and none of it is likely to be relevant to anyone who just wants to use the integration - so it can be safely skipped! If you’re interested in helping / understanding my pathology though, read on…
This is one of those areas where coming up with ideas is one thing but actually implementing them to find out if they work is quite another thing altogether!
The data we get is a set of “distance” measurements, which are:
- subject to the dark magiks of RF propagation.
- very noisy, with noise that is massively skewed to overshoots (ie, almost all measurements are actual distance plus a positive noise figure, a few might have near-zero noise, and a tiny amount might have a negative noise offset due to in-phase reflections).
- unreliably delivered. They might be late, or more likely missing altogether. A missing measurement is also indistinguishable on its own from an infinite distance measurement.
The unreliable delivery and asymmetric noise cause trouble straight away with the classical algorithms like Kalman filtering or even just plain averaging/smoothing. Luckily though, accurately putting certain things at the exact same place as a certain other thing, in as short a time as possible, is something that militaries are quite interested in, so there is (funded) research around, for those able to read the math. I am not skilled at the math, but I can sometimes grok the principles.
Re users drawing maps, that is a (long term) possibility, but I have no plans to do that any time soon. The main thing is that I think it will be unnecessary, but I’m in the wrong part of the Dunning-Kreuger curve to know for sure. I am (un?)reasonably confident that the system should be able to “solve” the spatial arrangement, and potentially-inaccurate user measurements would make that more difficult rather than less. I could be wrong, though. Ultimately I expect we’ll have our own “point cloud” which could then be “pinned” to a user’s map using translate/rotate/scale values.
My thinking is that if two proxies each measure the distance to one beacon, then the distance between the beacons can not be further than the sum of the two measurements (this is called “Triangle Inequality”). If the beacon moves around, that minimum-estimate will gradually improve, and if the beacon is placed at one of the beacons, we get our “answer” for the distance between those two proxies. From now on, when we receive measurements to that beacon, we have the lengths of three sides of a triangle from those proxies, so can arrive at two solutions in space for the beacon, relative to the proxies (the location is mirrored about the axis between the proxies). You can think about this by drawing circles - choose an arbitrary point as proxy-a. Draw a circle from that point with the radius of the distance between proxy-a and proxy-b, and choose anywhere on that as the location of proxy-b. Now if you draw a circle centred on each proxy for the beacon distance measured, those two circles should intersect at two locations, either side of the line between the proxies (or on the line, occasionally). In a way you have the “actual” position, and an ambiguous “reflection” of that position. The reason I think this is “better” than a user-defined map is that the distances then all come from the same, faulty system - rssi measurements, which I think might be easier to integrate than two different measurement systems. Some basic constraints from user input might be really helpful though, things like “these three proxies all lie along a single line, and this fourth proxy is on a line perpendicular to them, intersecting at the first one”, say.
When we have distance “measurements” between three points, we have solved the shape of that triangle but not which “reflection” of that triangle we have. If we bring in another proxy we get a third “fixed” point and solve where the beacon is in relation to those three points (in a 2D plane).
Once we achieve this, it stands to reason we can achieve this with every combination of proxy and beacon, at which point we have “solved” the spatial arrangement of all the beacons and proxies, relative to each other. At that stage one just needs to define a point in real space for any proxy, and an angle in real space to point at another proxy and the entire “cloud” is now aligned with real space. eg “consider my office proxy as being at [0,0,0], and the laundry proxy is exactly North of that”.
This is why I think we don’t need a user-supplied map to solve the space, and you can probably see that with the noisy nature of the data, trying to fit a user-dimensioned map adds a lot of complexity to something that hasn’t yet solved its existing complexities! 
I’m much keener for being able to visualise a 3D cloud of points in HA rather than going down the user-provided floorplan, but it would be a long time before I have time for either, I suspect.
So, being do-able for a single floor seems… waves hands likely? Solving in 3D for multiple floors is… less certain… certainly harder. I think that mathematically (as far as I can think mathematically!) there’s no reason why it shouldn’t work. With distances from two proxies you get a circle perpendicular to the plane (so it looks like two points in 2D), while distances from three proxies, if perfect, give you 2 points in 3D, which looks like a single point in 2D. Adding another proxy that’s not in the same plane (say, on a different floor) then gives us a hint at the “point”'s distance from the original plane. Except, none of these “points” are actually “points”, they’re all clouds of confusion, due to the noise and uncertainty. I think that adding that third dimension is likely stretching the already low-quality data a lot more than going to two did. I still think it’s possible, just laying out the difficulties so people can manage their expectations! 
As for at which point these uncertainties cause everything to fall apart (or simply be less than useful) I am not sure, but I am pretty confident that there is a lot of useful space between “not perfectly correct” and “entirely busted”.
My hesitation around solving for 3D is that while we work in only two dimensions, we sort of just fold the third dimension into the noise and try to ignore it (since any distance in the third dimension just causes an increase in all the measurements from the origin plane, which is exactly what RF noise/loss does, too). So in a way, extracting the third dimension is trying to extract meaningful data out of our left-over garbage. It should be entirely possible, but I suspect that this data will be even less reliable than that from a 2D solution - and I’m not entirely sure yet how reliable even that will be!
For now, I am going to stick to working out the 2D solution, because at this stage I believe 3d will need all the same solutions that 2D does, plus more - so it seems like a “progressive” path - we need to solve 2D first, and that work won’t be wasted in solving 3D, I think.
In practical terms, there’s probably a lot of functionality possible even without a 3D solution. For room-level occupancy, the nearest-proxy will still be pretty reliable (I expect typical inter-floor materials will add enough RF loss to make this easier) and combining with other sensors (PIR, mm-wave etc) will probably always be an important part, even for 2D (I am using Bermuda plus PIR for my office, and it’s rudimentary, but still great).
Perhaps grouping proxies by floor (which might be akin to what you were suggesting) in order to “constrain” each floor of proxies onto separate, parallel planes would then give us a per-floor location from which we’d choose by closest distance… I don’t know. It’s certainly something I am keeping in mind, but we are a long way from being able to worry about it in a productive way.
Re AI, I’m not sure how much benefit it might provide - again we’d need someone on the other side of the Dunning-Kruger curve to weigh in. Maybe it would be really good for tuning algorithms that clean up the distance measurements, or for real-time estimation of distances based on previously observed patterns - but maybe it’s not a good fit for any current ML patterns - I really don’t know. My gut feeling is that in these earlier stages there’s a lot to do in the basic mathematical realm of cleaning up and interpreting data, and that AI might be more useful down the road - but I could be wrong about that - maybe we could throw a sack of numbers into a GPU and it will work it all out! It would seem likely that it might be well-matched for estimating positions within a room given that RF reflections are so insanely complicated and simple at the same time. I’d love to hear from anyone who might have relevant experience in that area.
Sorry for the long rant, but part of it is just me mulling things over in my mind and thinking out loud.
My current thinking about next-steps that I can/will actually do are:
-
Beacon UUID / phone IRK/PrivateBLE matching
-
create per-proxy distance sensors for each beacon so that I can easily use the history tool to visualise what’s happening with the data, including missing packets. The current distance and area sensors are good, but their data makes it clear that
- missing packets are causing spurious area switches
- noisy data makes it hard to choose between nearby areas that are open plan (line-of-sight)
- having multiple proxies in one area really helps with both issues (graph showing (effectively) area-of-nearest-proxy and distance-to-nearest-proxy over time for my watch):
-
experiment with some smoothing/prediction filters on those distances to fill in missed packets and reduce the noise without losing responsiveness. I’m thinking Kalman filters, or maybe just a “two-euro” filter might be enough. Having the data easily graph-able right in HA will be a good start for evaluating different methods. I like the idea that a Kalman filter gives a “predicted” result, which might be particularly useful for making a decision when a given packet is missing. I know it’s sort of a low-pass filter that’s required, but given the noise is additive rather than symmetrical and also I want to preserve responsiveness I feel that simple smoothing might be a poor fit.
