I got my second vue utility since the old one isn’t working with both stock and esphome. This time I backed up the firmware so that I have an easy way to rollback to the stock firmware. esphome is still behaving the same way. I did get it paired with the utility and validated on the emporia firmware before switching over.
I have now switched back to emporia so that I can have a working instance, but like before, the device is no longer connected to the meter. I can’t tell what I am doing different compared to everyone else here, there is only so many ways to do this, and on both my devices, the mac address is correctly picked up, so esphome is talking to the MGM module.
The only difference I can see is that earlier in the thread, everyone had success with MGM Firmware V2 - my old device was V4 and the latest is V6.
[10:46:29][D][Vue:445]: Got meter join response
[10:46:34][D][Vue:496]: Sending request for meter reading
[10:46:39][D][Vue:496]: Sending request for meter reading
[10:46:44][D][Vue:496]: Sending request for meter reading
[10:46:49][D][Vue:496]: Sending request for meter reading
[10:46:54][D][Vue:496]: Sending request for meter reading
[10:46:59][D][Vue:496]: Sending request for meter reading
[10:47:04][I][Vue:503]: MGM Firmware Version: 6
[10:47:04][I][Vue:504]: MGM Mac Address: [..]
[10:47:04][I][Vue:505]: MGM Install Code: [..] (secret)
[10:47:04][I][Vue:506]: Trying to re-join the meter. If you continue to see this message
[10:47:04][I][Vue:507]: you may need to move the device closer to your power meter or
[10:47:04][I][Vue:508]: contact your utililty and ask them to reprovision the device.
[10:47:04][I][Vue:509]: Also confirm that the above mac address & install code match
[10:47:04][I][Vue:510]: what is printed on your device.
[10:47:04]**[E][Vue:511]: You can also file a bug at**
[10:47:04]**[E][Vue:512]: https://forms.gle/duMdU2i7wWHdbK5TA**
[10:47:04][W][component:214]: Component <unknown> took a long time for an operation (0.06 s).
[10:47:04][W][component:215]: Components should block for at most 20-30ms.
I got one of these recently and it has a MGM Firmware version 7 that sends a 44 byte packet instead of 152 bytes from version 2.
Spent some time and reverse engineered the payload enough to at least know how to get the values for Watt-hour Consumed, Watt-hour Produced, and Watts. I do have solar, so I can confirm it supports negative watts.
I’ve adjusted the header file to at least support reporting the Watts (I only care about this but might add reporting the consumed and produced Wh values later).
It’s at my fork here:
That said, this payload has separate fields for the consumption and production totals. So those of you who say that kWh Returned is always staying at 0, I believe somewhere in your 152 byte payload there is a field that is counting up the produced total and we just need to find it and use it. Happy to discuss the approach if anyone is interested in trying to figure it out.
Thanks @monkeyst . This is great work! My emporia vue no longer connects to the meter using both stock and esphome. I tried running your fork, but I am still stuck in the same state where the device does not talk to the meter. Your new logic for meter reading parsing is run after the connection is established, so I am unable to test this code. If I can get this working, I’ll try to run the packet capture script in your repo and confirm the schema for V6.
Hey, to be clear, the script I wrote is very rudimentary and doesn’t do the packet capture itself. You will have to use something else to save the sniffed data. I used a USB UART adapter and Realterm to save to a file, and Realterm supports prefixing each packet with a timestamp. A timestamp is required because the goal of the script is just to line up the packets with the kW reported by the Emporia exported CSV.
As for your problem, yeah looks like you aren’t getting any response back from the MGM111 chip when a meter reading is requested. That’d suggest the MGM111 chip has gone bad or the utility meter stopped working correctly… Are you saying the Vue works fine when you flash the stock firmware back on?
I am suspecting that either the MGM chip went bad or something needs to be hard reset with the stock firmware. I had it provisioned with PGE using the stock firmware, confirmed for 5 minutes that the data was being streamed and switched to esphome. It’s been broken since. I’ve now flashed back the stock firmware, but to reprovision again, I need to speak to PGE, which takes an hour. I’ll try again later this week and report back if I can get the MGM module working.
If the MGM module starts working after reprovisioning, I plan to capture the packets from the 3 pin header. I haven’t used Realterm before, but looking at the instructions, it’s fairly easy to setup. All of this is assuming that MGM module isn’t dead It’s a brand new device and has seen maybe 10 hours of usage.
I’ve flashed back and forth without needing to reprovision with the utility, which I think makes sense given the protocol is that the ESP sends a single byte instruction (+ header and footer) to the MGM. That is to say, there is no way the MGM would be able to tell that anything has changed about the ESP firmware and no way for a change on the ESP to affect the pairing status with the meter.
I think I have original Emporia backup. Ill post it somewhere for you. I would email Emporia support and ask for the MGM113 v2 firmware, and also maybe ask them if they can ask their dev team how to decode the MGM113 firmware packets.
Thank you. I had forgotten about this for the last couple of days. I got PGE to activate my new device again, and it turns out the meter had lost the pairing. Half an hour of back and forth and the new vue is on the network again, with stock firmware. The thing still keeps disconnecting periodically, often for days. The vue is plugged into the same location it was for the last several years and it’s well within 75 feet.
@monkeyst - what you say makes perfect sense and it is how I would have expected this to behave. I can’t tell what’s wrong with my setup. The location of the meter and the vue hasn’t changed, the meter is the same as for the last several years. There is some instability in my setup, and it is definitely not related to esphome.
@baudneo Thank you for the firmware. In case anyone’s interested, I have a V6 firmware backup that I took before I activated the device. Flashing this to my V4 device didn’t work - I assume it’s because the zigbee radio also needs a firmware bump to be able to talk the V6 protocol.
I think your correct in regards to Emporia fw and MGM111 firmware mismatches. It would seem to me that your power provider has a bug in their system or possibly a poor system implementation that keeps “forgetting” your device or maybe their database or software keeps crashing? I would ask your provider if any other customers have issues with any of the HAN devices. If other users with same/other devices have the same issue, it’s not on your end. Another idea, maybe your smart meter has a hardware/software issue?
I emailed Emporia asking them for access to the MGM111 firmwares and info on each firmwares flashing and MGM111 packet structure/byte length so we can decode usage data properly. I will report back if they do/don’t answer.
In the meantime I am also looking into the SDK for the MGM111. It may be possible to write our own firmware and then write custom .h file for esp to decode. I don’t know C well, so this may not work out. I’ll try and get a base first so other people who do know C can potentially help.
Hopefully Emporia helps us out, I told them the hass / esphome communities would appreciate it and buttered them up. So, fingers crossed.
Emporia replied and won’t help us at all. Looks like we need to deep dive.
Hi Tyler, thanks for reaching out to us. while we do think this is a cool use of our hardware, customers wishing to flash custom firmware on our products represent a niche corner of our market, and as such, we don’t have the resources at this time to provide you with the MGM chip firmware or the documentation you’ve requested.
Its like Steve-O said “I’m not mad, I’m just disappointed”.
Edit: SDK seems unlikely as zigbee smart energy requires a certificate that is handed out by the zigbee smart energy group and signed by their CA, from what I understand that costs a fair amount. This is good from a security standpoint as the group controls the single source of truth but a PITA for end users who want to play around.
My only idea is to reverse engineer each firmware versions packet structure. Hopefully they dont actively try and block our efforts here in the future. I will try and find a way to dump the MGM111 firmware so we can at least archive FW versions and possibly flash archived FW versions ourselves.
I’ve actually been getting no readings from the meter for several days now…
That’s interesting, I’ve had trouble with my PG&E meter as well and I wish I had your problem… my meter appears to be bugged out and says the Emporia is connected and receiving data when the Emporia is not even plugged into power. When speaking with the PG&E “supervisor” they said they don’t even have a mechanism to remove a pairing - they can only add pairings.
Emporia sent me a replacement and PG&E was unable to get that to pair, so in some way I need to get PG&E to take a look at my meter.
As for programming a firmware onto the MGM, on one of my calls with Emporia they did remotely push an update to my dongle and the MGM reports v8 now. I imagine the ESP must have been responsible for updating the MGM in this case, and perhaps the ESP firmware already bundles the corresponding MGM firmware and we would just need to find out how to trigger a “Program the MGM” routine on the ESP. Food for thought.
Aside from that, both chips have JTAG pins on the board, and there must be some sort of “esptool.py” equivalent for the MGM chip that we could use to manually backup/update it. Would need to be careful about not losing the signed certificate.
One theory I’ve had with why my meter bugged out is that the 5 second polling with the ESPHome firmware is too fast. For comparison, the poll rate on the stock firmware is every 30 seconds. Emporia’s product page says “5- to 10-second data interval” which is clearly false but would suggest to me that they found out 5-10 seconds was problematic and slowed it down to 30. I think the ESPHome version should follow suit so I’ve updated my fork to default to 30 seconds.
I have watts reported every 5 second polling via ESPhome and it has been ROCK solid. Never missed a poll and has never disconnected and had to be repaired in the 5 or 6 months Ive had it. The kWh sensors are every 5 minutes / delta: 0.1.
What type of meter do you have? I have the Landis+Gyr Focus AXR-SD with the GridStream RF protocol.
- name: "House Watts"
id: house_watts
unit_of_measurement: "W"
accuracy_decimals: 2
state_class: measurement
device_class: power
# Report every 5 s or when +/- 20 watts
filters:
- or:
- throttle: 5s
- delta: 20 # <- watts
- lambda: |-
if (id(fast_reporting)) return(x);
return {};
on_raw_value:
then:
lambda: |-
ESP_LOGD("Vue", "Watts = %0.2f", x);
Mine is a Focus AXR-SD as well, but sold under “Silver Spring Networks”, FCC ID: OWS-NIC514. Don’t know what protocol it uses but I know even when ESPHome worked and polled every 5 seconds, the wattage would only change every 10-15 seconds.
For the Wh readings, it seems to update its count as often as the wattage reading but I didn’t pay as close attention when I had ESPHome running. With the stock firmware, the Wh count would increase upon each 30 second poll.
I did this. It took 6 months, and I got a nice note telling me to pound sand: that code is proprietary and will not be released. Period.
I even tried to explain to them that allowing users to modify their device would save them money in the long run: they have a one-time (relatively small, compared to the BOM) income from each device, and an ongoing cost with S3 storage for all of the data. By allowing users to migrate to their own data storage, it would save them S3 storage costs ongoing. Apparently, financial arguments don’t work, either.
Still, keep asking. Eventually we might find an engineer on the inside willing to help.
From Emporia: Jared (Emporia Energy)
Jul 17, 2023, 11:05 MDT
Hi John,
So we do not have an open API and I can’t share any code with you at this time.
You can contact a few folks on github and see if they can share their reveresed engineer code or reports.
Please let us know if you have any other questions or concerns.
I’m surprised they would even suggest to look at reverse engineered code since that is effectively saying they are okay with customers opening up the device and changing the firmware. This is more permissive than I’d expect any company to be!
Convinced PG&E to replace my meter with a new one, now it is an AXRe-SD with FCC ID: OWS-NIC511-06 and I finally am getting HAN data again. Now I can confirm that V8’s payload is the same as V7 and I’ve realized that what I thought was 2 bytes each for the watt-hours consumed and returned are actually probably 4 bytes each. My repo has been updated accordingly.
I’ve also implemented support for reporting the kWh consumed/returned.
It’s a brand new device and has seen maybe 10 hours of usage.
