Stability problems NVMe on RPI 5

Installation Home Assistant OS
41

Wow, thats a lot you did and found out!

I finally surrendered and bought the RPI SSD and the official HAT because I was tired of further investing more time.

Your point with the signal cable may be valid! I was/am an embedded SW engineer and my gut feelings told me exactly that. When the problems got worse here I suspected that cable. So I disassembled everything, checked that cable, turned it on again and the problems visible during startup were all gone.

Initially I was happy but already after closing the housing/fixing the screws and putting the RPI to its right location it did not even boot anymore. So for me this just looked like a problem regarding signal quality, depending on unknown factors. So I strongly believe in your theory.

Another finding I had was: during this time I had a display connected to the housing, so I could monitor the logs from time to time. I never connected a display before. But now my RPI had suddenly massive trouble connecting to the WiFi. As soon as I detached the display cable again everything was fine again.

My lesson learned from this whole thing is, that I would never use an RPI again for this purpose. We want to have a stable HA and it is extremely annoying to explain your wife why the bathroom remains cold.

Taking all the money I invested in new PSUs, HATs, SSDs I could have afforded a most likely more stable Mini PC with even some money left, taking my wife out for dinner, aka saying sorry for the cold bathroom :wink:

42

Can you have a look at what this returns in the CLI?

The issue here is much much bigger. I found out on my search for answers.
There is an incompatibility with many NVME SSD and raspberry pi 5 but not only with RPi5 It also pertains to other platforms and hardware systems. It has to do with Phison NVME controller. That causes the issues.

I was experiencing shutdown crashes and freezes moving from 15.2 to 16.1 and up to the point where it drove me crazy. I ended up completing wiping HA of my NVME and making a new install only to find out that did not help. Digging further it turned out that all the NVME’s I have SABRENT Rocket plus 1T, Kingston NV3 WD350 all have a poison controller which causes these issues.

I ended up grabbing a 256gb USB3.1 stick dongle thingy and running HA on it. After a restore I luckily recovered my instance. and it has now been running smooth for 36 hours.

more to follow.

https://www.google.com/search?q=nvme+ssd+incompatibilty+with+rpi5+phison&client=firefox-b-d&hs=yWIp&sca_esv=89aaa52b873f06b6&biw=1710&bih=989&ei=Y_yEaeOhD5mBi-gPl4-C-Qw&ved=0ahUKEwij34Whl8OSAxWZwAIHHZeHIM8Q4dUDCBM&uact=5&oq=nvme+ssd+incompatibilty+with+rpi5+phison&gs_lp=Egxnd3Mtd2l6LXNlcnAiKG52bWUgc3NkIGluY29tcGF0aWJpbHR5IHdpdGggcnBpNSBwaGlzb24yCBAAGIAEGKIEMggQABiABBiiBDIIEAAYgAQYogQyBRAAGO8FSMAqUNAFWJcdcAF4AZABAJgBhwGgAaEFqgEDNC4zuAEDyAEA-AEBmAIIoALPBcICChAAGEcY1gQYsAPCAgcQIRgKGKABmAMA4gMFEgExIECIBgGQBgiSBwM1LjOgB9YcsgcDNC4zuAfHBcIHBTAuMy41yAcXgAgB&sclient=gws-wiz-serp

43

@330chauf you are right that the Phison NVMe controller is know to be a problem but in fact, it generally prevents the NVMe SSD to be recognized by the Pi. You can install on it or boot it.

As far as I know, the Patriot is not relying on this controller so I don’t think it is the issue. At least I have not seen the Patriot on the lists I found of problematic NVMe SSD, in particular the Geekworm one.

36 hours is a start but it is not a lot… As for me, without any configuration change, my RPI 5 has been running for a week now. The only change I can think about is that I gave up with WiFi (because of the problems during reboot as experienced by others and mentioned recently by @dan-shaqfu ) and decide to connect the RPI 5 to the network with a RJ45 cable after the last reboot and 1 day out of the network because of the WiFi issue. Since then it has been stable but I think it doesn’t make sense to correlate the 2 things…

I decided to wait a little bit more before doing new checks and rebooting the RPI 5…

Still tuned! But I am not really satisfied by the situation suddendly becoming stable without any reason…

44

Could be that it has no effect. But I let it run overnight with raspberry Pi lite overnight. Meaning booted but no running extra processies and sure enough in the morning there was no connection anymore.

I’m consediring moving my installation to a NUC. Speed is one thing but unreliability is something totally different

45

Apologies. Reading again my previous post, I realized that I wrote the opposite of what I wanted to say at the end of the first paragraph… Should read “you can’t install on it or boot it”…

I also wanted to report that my RPI 5 is still running like a charm and that thinking again at the possible cause, I consider more and more seriously an impact of the WiFi on the SSD. It is really the only change: since the last boot, the WiFi interface is down, without any way to restart it except a reboot that I don’t want to do for the time being (for testing purpose)…

For what is worth, I decided to ask ChatGPT and I got an answer saying it was not completely a crazy idea. My question was: on a rasperry pi 5, could the WiFi have a negative influence on PCIe signals ? Below is an except from the (long) answer:

Why Wi-Fi could affect PCIe

Wi-Fi (2.4 GHz / 5 GHz) and PCIe are very different beasts, but they can still interact indirectly:

EMI (electromagnetic interference)
Wi-Fi radios transmit RF energy. If PCIe traces, cables, or connectors are poorly shielded or routed, that RF noise can couple into the PCIe signals.

High-speed sensitivity
PCIe Gen 2/Gen 3 signals are extremely fast (multi-GHz). They don’t tolerate noise well if:

Signal integrity is already weak

Eye margins are small

There’s impedance mismatch, reflections, or long cables

Ground / power noise
Wi-Fi transmission draws bursty current. If power delivery or grounding is marginal, that noise can show up as jitter on PCIe lanes.

In particular, in my case I use the Geekworm X1001 PCIe HAT which seats on top of the RPI instead of under the RPI with the official HAT. As a result the FCC cable is longer, meaning it is more exposed to signal perturbations from the environment and WiFi may be one of the source of such a disturbance… I have nothing to prove it, just an idea…

At next reboot, I plan to so some tests reactivating the WiFi to see if I can reproduce the unstable sitatution and stop it by disabling it. If it is the case, it is an easy workaroud for me as using an Ethernet cable is not a problem (was just easier with WiFi).

46

These settings absolutely saved the day for me with a Raspberry Pi5, Argon V3 enclosure, Raspberry Pi5 power supply and a Ranxjiang (yeah, never heard of them either) 256Gb NVME

47

Thanks for this topic. It helped my HA on Rasperry Pi 5. I share my situation and result.

Hardware setup

  • Board: Raspberry Pi 5 (8GB)
  • SSD: Crucial P310 1TB NVMe (PCIe Gen4, M.2 2230)
  • NVMe HAT: Generic simple M.2 NVMe PCIe HAT, NoName X1001
  • Power supply: Official 27W USB-C PSU
  • Metal Case

Symptoms before fix

  • Random system freezes every 1–3 days
  • Filesystem suddenly remounted read-only
  • UI randomly broken (frontend JS loading errors)
  • Recorder + MariaDB failures
    I found a lot of errors like:
OSError: [Errno 30] Read-only file system
OSError: [Errno 5] I/O error

Fix way
Since Home Assistant OS does not expose /mnt/config.txt directly, I did this by:

  • Boot Raspberry Pi OS from microSD (SSD is still connected). It need to be boot-settings in raspi-config: microSD → SSD/USB
  • Edited the file /mnt/config.txt on the SSD. I added:
dtparam=pciex1
dtparam=pciex1_gen=3
dtparam=pciex1_aspm=off
dtparam=pciex1_msi=off
dtoverlay=cma,cma-96

Remove MicroSD and boot HA from SSD.

Before this setting my SSD worked on PCIe Gen2:

  • dd write: ~400 MB/s
  • dd read: ~420 MB/s

With “dtparam=pciex1_gen=3” it work on Gen3:

  • dd write: ~740 MB/s
  • dd read: ~820 MB/s

Result:

  • 2+ weeks uptime without a single freeze
  • No filesystem remounts
  • No recorder / MariaDB errors
48

I’m new to the thread but followed it for a bit while troubleshooting similar issues on my side, with each passing day I believe more in the wifi theory.

I was having nvme issues for 2 weeks where sometimes it would fail on boot, or 20 minutes after boot.
Stopped the docker container running homeassistant and disabled wifi, had like 2 or 3 days with no issues, so I assumed it was fixed by either stopping home assistant or disabling wifi.

Started home assistant container again and it has been 3 days without issues, the maximum I was able to achieve before was like a little over a day of uptime.

This whole time where I’ve had no uptime issues the wifi has been disabled, so either I’m just “getting lucky” or wifi was indeed the cause.

49

Thanks a lot for all your input, it seems, this solved my issues.

Hardware setup

  • Board: Raspberry Pi 5 (8GB)
  • SSD: ADATA Legend 960, 1TB, (M.2 2230)
  • NVMe HAT: Geekworm X1002
  • Power supply: Official 27W USB-C PSU
  • Metal Case

Symptoms before fix

  • HAOS 17.1 did not install, process crashed with IO-errors
  • Raspbian and Ubuntu installed and ran on the same SSD without issues without mods in config.txt

Fix
Added just dtparam=pciex1_gen=3 to /nmt/config.txt after booting Raspbian from SD-card

Result: I was able to install HAOS 17.1 and import HA backup from my older RPi4. System runs now for several hours without issues.
I’m really confused that raspbian and Ubuntu worked fine with default settings and just HAOS did not.

50

Hey Fix, Thanks so much for the tips.

Especially the part on writing config.txt on the HA SSD.

For me it’s stable for the first 24hours even on an SSD which should not reallt be compatible with the ARGON NEO

Keeping my fingers crossed but it does look promissing for now.

51

@pe-carmo good to here you were able to reach a stable state with WiFi disabled. As for me, I had no issue since I don’t use anymore the WiFi and I tried to reenable it for testing and it was a disaster.

I don’t say it is the only source of problem but I think that if you use a HAT X1001 (and probably any other HAT seating above the Pi), you should not activate the WiFi or you will be exposed to the symptom described in this thread.

As for the kernel parameters, I think the only one really needed it the one to use Gen 3 mode. In my test, using Gen 2 tended to be a disaster too (with the Patriot NVMe that seems to work very well in Gen 3).

To edit the kernel configuration, you should not need to boot on another device if you are lucky enough to have 1/2 hour without the problem and are a little bit familiar with Linux tools. When on the console, just type login and you will get a shell prompt where you can use vi (if you know how to use it) to edit the file which is under /mnt/boot.

52

The WiFi theory is interesting. I already suspected some signal noise as root cause and I had issues related to WiFi.

However, I just wanted to add that I did use WiFi as well but got rid of the NVMe problems after the hardware change.

But, I had more and more trouble having a stable WiFi. I monitored the Ping delay/quality to 8.8.8.8 and sometimes, especially when using 5 GHz it was really really bad. My RPI was the only device having such issues.

In the meantime I switched to a wired connection and finally I am happy with my setup.

Hope this helps somebody

53

Just wondering what the resonant length of some of the conductors/PCB wiring traces carrying disk data is in relation to wireless signals in very close proximity, both at 2.4Ghz and 5Ghz wavelengths? For the X1001 as well as Raspi Hat versions, and possibly others, commonly found in Argon cases.

Maybe this entire issue is just a simple case of poor RF design and shielding for the SSD connections and HATs being swamped by WiFi, right there, millimetres away, corrupting the disk data, and the disk data radiating from the traces, corrupting the WiFi.

The drive ECC process has probably been getting some serious workout.

Anybody done some actual real life measurements and connected a scope and signal analyser?

54

Poor RF design is exactly what I suspected. The effects we’re so random there was absolutely no good explanation… I used to work as an embedded engineer building radar related HW/SW. The RF topic is just insane. Even skilled engineers try fixing issues “with gut feelings”, like adding a cap to ground here and there…

Regarding WiFi I just wanted to add a diagram showing min, max, avg Ping latencies to 8.8.8.8 correlated to the band in use. This version is a good one although the difference is obvious. I had 5Ghz connections (maybe a different channel) causing outages and basically prevented using the Pi at all.

23892
238922000×337 80.3 KB

55

Just wanted to report that it is now 1 month and half that the RPI5 has been running very smoothly, without any issue, since I identifed WiFi as the cause of the problems and I disabled it. I understand that it is not always possible to disable (not configure) the WiFi but it you can and you are suffering the kind of issues described here, I think it is an easy workaroud.

I also agree that the root cause is a RF design issue. I tend to suspect that the HAT (X1001) is the culprit but I had no opportunity to try another HAT.

56

As i’m also victim of the unstable RPi5 hassio, what is the way to disable WIFI in hassos?

My machine (Argon Neo 5) runs pretty in idle on a test bench for weeks. But if installed as workhorse, with Z-Wave, Zigbee, Grafana after few days the filesystem crash into pieces.
Bluetooth is mandatory for Switchbot devices, but i have a LAN cable so WIFI should be switched off, if this is even possible in hassos.

I bought 3 different hats and all fall into the same problem, i bought 4 different NVME and they did the same. Last choice was the Argon NEO 5 case with their script setup, but 18 days later all went down again.

I also would like to raise the hmb for the NVME drive, found out that we eneed 96MB and hassos only reserves 64MB. The github issue about that was wiped aside. It should be configurable in config.txt, but yes we are on hassos where this file is firewalled as hell. No way to reach it over SSH and sure it will be reverted with any hassos upgrade after change.

57

it seems you invested a lot of money so far :wink:

as some others already commented, using a miniPC for this purpose is much cheaper and probably more stable.

However, I think I cannot answer your question since I never disabled wifi completely. But there should be some public resources available which might be helpful.

I switched from wifi to wired lan (leaving wifi enabled as a backup). This did not solve my NVMe issues, but the network issues I had. The problem regarding the NVMe was solved here, after installing the “official” raspi HAT + NVMe.

hope this helps…

58

A friend working in the electronics looked at the pie and hat cable, told that the cable should be shielded to prevent shaked bits while transfer.

My Argon One now runs over 1 week without problems, BT is on while WIFI is disabled. I hope this will stay solid, but yeah i also thought about something like a refurbished Futro. But the ARM64 base would consume more energy than a ARM64 platform. Maybe this will come to fruitition, if the Argon fails me again.

59

@LinuxfarmerHH thanks for reporting these tests, even though they are not encouraging... About the various HAT you tested, were they all sitting on top of the RPI (like the Geekworm X1001) or was one of them placed under the RPI (as the official HAT or the Geekworm X1002)). In a previous post, I made the hypothesis that the "antena issue" may be reduced with a HAT under the RPI, I'm interested to hear feedback from those who tested this other configuration...

60

A bit late,

but yes the different hats are all placed with distance bolts above pie. The Argon Neo was my first with hat below the pie. It did only crashed once, since deactivated WIFI it's stable. BT seems not to be a problem for the hat cable, i require that for Switchbot

Now with 30° in the room only 43° CPU without running fan.

LinuxfarmerHH