GPIO 16 was taken from a Blink example at ESP32 WIFI/BLE Board v1.0 - Elecrow Wiki. There are also Eagle design files at the bottom of the page but I am unable to open them on Linux (plus I am not really sure if they’d be of any use).
In the meantime I ordered an 18650 with a JST-PH 2.0 connector to see if it resolves the problem.
Let’s hope so. Be aware that most of Esp-boards with battery connector have charge controller but not over discharge protection. So if you run the battery until it dies, it will eventually get damaged. Only protected batteries are safe in this case.
I got a portable power supply and found out that somewhere between 3.97 and 3.98 volts the board stops/starts to work. I can do this repeatedly: with output set to 3.97 V the board fails to load/run the code, with 3.98 V the blink program works as designed.
This likely explains why it was running fine with a fresh battery (charged above 4 V) but did not last for long. I guess I’ll need to look for another board 
Ridiculous board, at 4V you still have 90% of battery capacity left.
This may be an aging problem. The board is several years old and most of the time it has been sitting at the bottom of a drawer.
No way. Battery will degrade in drawer, dev board will not.
There is also the possibility that the ESP doesn’t have enough power rail decoupling, and/or the battery board includes a DC-DC converter that injects more noise than the USB interface.
Early ESP12F were bad for this, and even with a load of 100nF + 10uF caps across the power rails, still could restart randomly. I would expect more modern ESP32 boards to be better, but this could be a bad design.
You’ll not spot RFI without a storage scope, but a hack is an AM radio tuned to static. Placed next to the dev board, you should be able to “hear” the rhythm of the processor and get a rough feel for any differences between USB and battery power.