web_serial
v1.0.0 – field-hardened. Both platforms bench-tested at length: ESP8266
(80 and 160 MHz) through a six-crash forensic campaign that ended with the
umm/lwip SYS-context allocator collision identified from decoded HWDT stacks
and closed (ALLOCQUIET + a cohabitation valve); ESP32 (Arduino and IDF, 16
ports, 4096 B buffers, M=15 loopback storms at 90+ %% engine). Killer-config
uptimes went from 27-65 s to storms outliving the bench session. Full war
journal: docs/BUILD_STATUS.md; every trap harvested: docs/PIEGES_MATERIEL_ESP8266_ESP32.md.
It remains what it is: a LAN bench tool, no auth, one operator.
An in-browser managed SERIAL SWITCH for ESP8266 and ESP32. Like an
Ethernet switch, but for serial: ports (physical UART, web console, TCP, UDP,
PC-serial bridges) are grouped into VLANs – a byte in on one port reaches
every UP port in the same VLAN, each at its own speed. Ports are created at
runtime from the web UI, with a memory guard that refuses any allocation
that would drop the largest free block below the platform safety floor. Fourth of the family (web_i2c, web_spi,
web_onewire) – same transport, same log/decoder discipline – and the FIRST
with no build-time graft: the uart debug callback is a public runtime
seam.
The hub’s ports
| port | nature | class |
|---|---|---|
| physical UART | the ESP’s real serial pins | QoS |
| firmware (host) | other components on the same uart: bus |
native QoS bypass (zero code, zero loss by construction) |
| web frontend | text+hex console, tap log, decoders | best-effort, lag-tolerant |
| TCP232 | raw network serial – PuTTY, telnet, Eltima Splitter direct | write=QoS path, read=best-effort |
| PC serial bridge | the browser opens a LOCAL COM port (Web Serial API) and joins the hub | best-effort |
QoS doctrine (the sketch’s yellow marker): the physical<->firmware path loses
NOTHING – it is ESPHome’s own native path, merely observed. Web/TCP/PC are
observers-and-talkers that may drop under load (flooded badge, honest
banners), never at the main path’s expense.
Protocols and physical layers
- Modbus RTU – silence framing (Modbus preset: 4 ms), decoder with slave
address, function names, exception frames, and the CRC16 verified
in-browser (poly A001, wire order LSB first). - DMX512 – 250000 8N2 preset; RX frames decoded (start code, channel
count, first 32 channels as live bars). TX is experimental: the BREAK is
generated by the baud trick (one 0x00 at 90000 baud = ~100 us low), then the
frame at 250000. Good for bench fixtures, not for show-critical timing. - NMEA 0183 – sentences named, XOR checksum verified. AT dialogs
spotted. Everything else: honest hex + ascii dual view. - RS485 – half-duplex: give
de_pin:; the hub asserts it around every
write and releases afterflush()(MAX485-style DE+/RE- tied together). - RS422 / plain TTL – electrically transparent, nothing to configure.
Install
external_components:
- source: github://kaboom748/web_serial
components: [web_serial]
refresh: 1d
uart:
id: ser_bus
tx_pin: GPIO17
rx_pin: GPIO16
baud_rate: 115200
web_serial:
port: 8891
uart_id: ser_bus
tcp_port: 2323 # optional raw TCP232
# de_pin: GPIO4 # RS485 driver-enable
owner: true # false if ANOTHER component reads this uart (tap mode)
gap_ms: 10 # silence closing a frame
# heap_floor: 6144 # override the safety floor (bytes); lower it, at your own
# risk, on a RAM-starved ESP8266 sharing many components
Several hubs? Just instantiate several web_serial: blocks (MULTI_CONF), one
per uart. Open http://<ip>:8891/; point PuTTY at <ip>:2323.
The managed switch
- VLANs per port – assign each port a VLAN (1-8); ports only forward
within their VLAN. Isolate two VCOM sessions, or wire a monitor-only group. - Dynamic ports –
+ TCP/+ UDP/+ Bridgein the UI (or
PORT ADD TCP 2324 512) create ports live;PORT DELfrees them. Up to
8 ports on ESP8266, 16 on ESP32. - Port up/down – admin shutdown per port (blocks both directions).
- Storm control –
PORT RATE <id> <bytes/s>caps a port’s ingress; the
excess is dropped and counted (a red dot flags a tripping port). Blind to
slow gain-1 loops by design: that is loop detect’s job (below). - Loop detect –
LOOPDETECT ON|OFF|KILL(default ON), fully PASSIVE:
zero bytes are ever injected in the stream, on any port. The hub
fingerprints its own egress per network port (FNV-1a + length, 4-slot
ring, ~40 B/port); an ingress chunk matching a <=3s-old fingerprint from
a port of the SAME vlan, three in a row, means the frame came back where
it started: recirculation, a loop. Cross-port loops (tcp<->bridge piping)
included; badge and KILL land on the ingress port. The UART is exempt
(the TX->RX jumper is a deliberate echo).PORT UP/PORT RESET/ a
vlan change clears the badge. - Inter-VLAN bridge detect –
XVLANDETECT ON|OFF|KILL(default ON),
same passive engine, other verdict: an ingress matching recent hub egress
from ANOTHER vlan crossed once and died there – not a loop, a bridge
between vlans, i.e. broken isolation. OFF tolerates deliberate bridging
in total silence (the disabled pass neither counts nor consumes
fingerprints, so a live loop elsewhere still gets caught); ON badges
XVLAN+ logs; KILL downs the bridged-into port. Both detectors are
independent – all four combinations work. A true circular loop through
two vlans is attributed to LOOP on the re-entry ports, not XVLAN. - Detection limit, accepted and field-verified – both detectors match
EXACT chunks (hash + length). Large blocks that the transport re-chunks in
flight – the bridge WS framing (64 B), the hub’s own TCP reads (64 B),
OS/driver COM coalescing, or egress-buffer eviction under pressure – come
home in different pieces and are deliberately NOT flagged: a miss, never a
false alarm. Short chunks (a lone CR+LF is enough) traverse atomically and
are caught within three passes, and a re-fragmenting loop still screams in
the counters (symmetric tx/rx across ports, climbing drops). A
sliding-window (Rabin-Karp) mode was considered and rejected to keep the
zero-false-positive contract. - Hot vlan changes –
PORT VLAN <id> <v>(or the VLAN pill) applies
live; it also purges that port’s detection state (ring, streaks, badges):
a vlan change is a topology change, detection restarts from zero and the
3 s stale-fingerprint window is closed. - Multi-port PC bridge – up to 4 local COM ports via the Web Serial
API, each bound to its own hub bridge port. Both WS directions carry the
bridge slot id (0x01 slot dataup,0x02 slot datadown); the page
demuxes per slot, and disconnecting (or unplugging) a port sends
PORT DEL– no orphaned bridge ports, no duplicated egress. The
+ Bridgebutton now opens the same connect flow (a bridge without a
COM port behind it is a dead end). - Per-port counters – TX / RX / drop, live; the Reset counters button
(orPORT RESET) zeroes them and clears LOOP/XVLAN badges. Each row
also shows a live rate pill (storm-control cap, click to edit) and the
egress buffer size on hover; Line settings, owner/tap mode and the
wire-tap visibility state all round-trip from the hub, so a page
reload always shows the LIVE configuration. - Console VLAN – the web console talks on the VLAN you pick (
CVLAN). - UDP for VCOM – a UDP port learns its peer from the first datagram
(matches VCOM’s UDP mode). Works on BOTH platforms: ESP32 through the
BSD socket layer, ESP8266 through the bundled wser module (raw udp_pcb,
lwip-only dependency, gauntlet-tested standalone: 8 suites, byte-exact
100k-op fuzz, ASan/UBSan, 6/6 mutations caught – see wser_*.{h,cpp}
and the PIEGES-compliance notes inside). Same behavior on both: peer
learned from the first datagram, txerr counter, self-recreating socket
after 5 consecutive send failures, 5s/3-strikes bind backoff.
DESIGN DECISION (learning-only, no configured target): the hub never
initiates – the UDP client MUST speak first, the hub learns the peer from
that first datagram and replies there. Consequences, accepted: a fresh UDP
port in a busy VLAN drops egress until the first datagram arrives; a
hub-to-hub UDP trunk is impossible (two pure learners never start –
ESP-NOW is the answer for that); and last-talker-wins – any LAN sender
can steal the peer, fine for a bench tool. Field bug fixed on the way:
peer storage is now a real (aligned) sockaddr_storage – lwip’s sendto
rejects misaligned sockaddr pointers with EINVAL, which made every reply
fail forever while ingress worked (the climbing-txerr signature). - Persistence – dynamic ports, VLANs, the console VLAN, per-port shapers
(rate/out), buffer sizes, loop/xvlan detect modes AND trunk/hairpin wiring
(wire) are saved to ESPHome preferences and restored at boot (your
startup-config). NOT persisted: the liveLINEsettings – the YAML uart
config wins at every boot (runtime baud changes are session-only; making
them sticky is an open design decision).
Memory guard (heap / largest block / fragmentation)
The defense onion (who fires first, who sleeps – health = outer layers
work, deep layers sleep): largest < 7 680 → governor blue tick (quota /2);
< 6 144 → cohabitation valve (one lwip submission per pass, info at 3.5 s);
< 5 120 → radio brake (hard); < 1 536 → tap guard (frames dropped, cause
heap). Field reference: blue ticks with drop causes heap = 0 = the
outermost layer absorbing everything. YAML-side constraints and their
observable signatures: see CONFIG_ESP8266_TERRAIN.md.
The side panel graphs free heap, largest contiguous block, and
fragmentation %% live. The largest block is what actually matters on ESP8266:
the heap can read 30 KB while the biggest allocatable chunk is 4 KB. Before
allocating a port buffer, the guard checks that the largest block would stay
above the floor (8 KB on ESP8266, 20 KB on ESP32) – if not, the
allocation is refused with a message, never a reboot. You choose the
buffer size per port (256 B default / 1 KB max on 8266; 512 B / 4 KB on
ESP32); an inline preview shows whether it fits before you commit. The floor is
adjustable live (the Floor field, or FLOOR <bytes>) and can be set in YAML
(heap_floor:) – on a heavily loaded ESP8266 where free heap is already
near 8 KB, lower it deliberately or move web_serial to its own chip.
owner vs tap (the destructive-read rule)
Reading a UART CONSUMES bytes. In owner mode the hub reads RX itself and
is fully interactive. If another component (modbus controller, a PZEM…)
owns the reads, set owner: false: the hub then only OBSERVES through the
debug callback (their reads and writes appear in the log), and hub writes are
still possible but may interleave with the owner’s protocol – your call.
Live line settings
Baud / data bits / parity / stop bits change from the page through ESPHome’s
official load_settings() – no reflash, with DMX (250000 8N2) and Modbus
(4 ms gap) one-click presets. Framing gap and optional delimiter (0A for
line protocols) too.
The PC serial bridge (Web Serial API)
Chrome’s navigator.serial lets the page open a COM port ON YOUR PC and
bridge it into the hub: PC-port RX goes onto the ESP’s wire; the wire’s RX is
written to the PC port. Binary WS frames, 0x01/0x02 tags. Secure-context
caveat: on plain http://<ip> Chrome hides the API – enable
chrome://flags/#unsafely-treat-insecure-origin-as-secure for the ESP’s
origin (documented, honest limit).
The web UI – three tabs
General: the switch table (VLAN pills, up/down, rate/out shapers, wire
pills, LOOP/XVLAN badges, live per-port meters), send-on-the-wire, the log,
SERIAL DECODE, and the vlan terminal (below). UART: line settings, presets,
the ESP8266 pin-constraints map. Views: eight live cards – TOPOLOGY ring
(click a node for its blast radius), the GOVERNOR AIMD chronogram with
cause-colored collapse ticks (red backlog / amber duty / blue heap, all three
field-captured), LIVE MODEL (a continuous regression test: measured vs
bottleneck-law prediction, the exact multi-vlan Sigma-I law, a cycles/byte
cost line and the service ratio that names duty-broken regimes), FLOW &
CONSERVATION (byte-exact at every regime the bench produced), INTER-ARRIVALS,
MEMORY MAP with the three floors, TAP FUNNEL, and ENDURANCE (uptime, valve
firings PER HOUR, the retained black-box line, largest/frag vigil).
The LOGGING knobs (Detail Full/Summary, Delivery Live/Batch/Off, Filter)
govern the LOG stream only – never the wire, never the counters, never the
terminal. Full+Live feeds the decoder and the histogram; Summary/Batch are
the operator’s own valve under storms. Details: docs/GUIDE_COMMANDES_web_serial.md.
The vlan terminal – the console that cannot lie
The RAW panel is a byte-true mirror of everything the vlan delivers to the
console member, fed by its own ring + a dedicated binary stream under the
same egress valves. Its contract: loss is possible, lying is not. Any
ring overflow (counted on the console port’s own drop) travels IN the
stream as a confession record and prints inline as a red [lost N B]
marker you cannot copy over; the header banner states vlan N raw -- in-sync or LOST N B at all times; every send and file/clipboard transfer
closes with a delivery receipt from the members’ real counters
(delivered -> u0 +12512 (drop +32) | t2 +12512). Party-line doctrine:
one neutral tint for ALL data (a byte is a byte – com, bridge, uart, tcp);
color is reserved for instruments (lost=red, receipt=blue, local echo=gray).
Scope = the console’s vlan, exactly like the wire tap: move the console to
see another vlan.
Documentation
docs/GUIDE_COMMANDES_web_serial.md– every command, every instrument,
reading the colored ticks, the terminal contract, the LOGGING knobs.docs/CONFIG_ESP8266_TERRAIN.md– each YAML constraint with its
OBSERVABLE signature (power_save blue ticks, f_cpu, safe-mode economics,
the defense onion table).docs/PIEGES_MATERIEL_ESP8266_ESP32.md– 56 field traps, each paid for.docs/LESSONS.md– the distilled engineering lessons.docs/BUILD_STATUS.md– the verbatim campaign journal (the war log).
Known limits (assumed, not bugs)
- LAN tool, no auth, one web client + one TCP232 client (a new one replaces).
- Simultaneous talkers interleave on the wire (it is a hub, not a switch).
- DMX TX timing is approximate (baud-trick break); DMX RX depends on the UART
swallowing the break as a framing artifact – gap framing still splits
frames correctly at 4 ms. - ESP8266: UART0 belongs to the logger by default (set
baud_rate: 0on the
logger or use swapped pins); UART1 is TX-only. - Web Serial API needs Chrome + the insecure-origin flag on plain http.
Host tests (8 suites, one command)
bash tests/host/run_all.sh
Six C mirrors (crc16, framing, switch incl. the cons-ring truth
contracts, guard incl. the valve arithmetic, loopdet, shaper), the
decoder JS mirrors, and a 177-assertion UI suite that extracts the real page
out of PROGMEM and drives it in jsdom – badges, pagination merge, the Views
model, the terminal’s loss markers. CI runs the same script plus a three-target
compile matrix (esp8266 / esp32-arduino / esp32-idf).
Hardware sign-off and the full campaign journal: docs/BUILD_STATUS.md.
UART on ESP8266 – the constraints map (mirrored in the in-app guide)
Three hardware combos ONLY – UART0 (GPIO1/3, USB-shared), UART0-swap
(GPIO15/13, the full-duplex choice; move the logger to UART1), UART1
(GPIO2, TX-only by silicon). Anything else = silent SOFTWARE bit-bang:
~57600 practical, sustained TX can wedge the chip under WiFi load (this
hub shields its egress to ~2 ms/pass there). Strapping rules: GPIO15 low
at reset, GPIO0 low = flash mode, GPIO2 high; never GPIO6-11/16 for
serial. DMX (250000 8N2, exact divider on the 8266): frames are 513 B –
rx_buffer_size: 1024. A build-time audit prints the verdict in the config
dump and the UI (UART row: green hardware / amber software), the egress
quota law follows it (wire-saturating on hardware – measured 1918/1920
B/s at 19200, 14.7 kB/s at 250000 – shielded on software), and the info
panel shows rx_buffer + TX FIFO. ESP32: pin matrix, any pins, 5 Mbaud;
GPIO34-39 are input-only.
Traffic management (per-port shaping + live instruments)
Every port has TWO independent rate pills: rate (ingress storm control,
unchanged) and out (NEW egress shaper – a lazy 1 s token bucket at the
drain; excess overflows the port’s buffer, evicts oldest, and counts in
drop, exactly a real switch on a speed mismatch: cap the uart at out 9600 under a 19200 flow and watch half the bytes die AT that port).
PORT ORATE <id> <bps> (0 = unlimited), persisted (config magic v4 –
saved ports reset ONCE on first flash of this build). Live bars per port
(4 Hz bs telemetry, snprintf-on-stack, gated by client/backlog/heap):
OUT fill for every buffered port, IN fill where a real buffer exists
(uart ring, wser udp ring), an honest flow-ctl label for TCP (its
backpressure lives in the TCP window, not in a local buffer). SYSTEM
gains three instruments: Peak (session high-water of tx+rx), Engine load
(duty-pool utilisation %), and Headroom est. (throughput/load – labeled
as the estimate it is). The uart bar row also states the wire’s exact
ceiling (baud/10).
TRUNK – the virtual wire (declared inter-VLAN bridging)
Two slot-less bridge ports can be WIRED together into a RAM null-modem
inside the hub: PORT WIRE <a> <b> (or the wire pill), PORT WIRE <a> -1
to unplug. Egress into a wired bridge ENQUEUES; a governed drain (same
fairness cursor, same duty budget, <=256 B/pass) injects into the peer’s
ingress. The out pill IS the wire’s baud – shape the trunk like any
port. Put the two ends in DIFFERENT vlans and you have a legitimate,
counted, shaped inter-VLAN trunk: the crossing is DECLARED, so XVLAN
detection exempts it (green XVLAN ALLOWED badge on both rows) while
still hunting clandestine paths; LOOP detection stays fully armed around
it. The origin rule traverses the wire (no ping-pong inside the pair).
Persisted (config magic v5 – saved ports reset once); load-time
validation unplugs any asymmetric/invalid pair. Related instruments:
hbrake per-port counter (bytes dropped by the radio heap brake – every
extinguisher signs its catches) and RADIOFLOOR <n> (tune the brake’s
floor; 0 disables it with a red warning – documented FIQ-wedge risk;
not persisted, re-arms at reboot).
Session additions: hairpin, meters, operator handles
HAIRPIN: PORT WIRE n n (or the wire pill with the port’s own id) turns a
slot-less bridge into a declared LOOPBACK – its egress re-enters its own
ingress through the governed drain: the integrated loopback plug, a
one-port storm generator (the out pill is the oscillator’s period), an
echo responder, and the pure-hub benchmark target. Cyan \u21a9 LOOPBACK
badge; the LOOP detector will light on it and that is the witness telling
the truth about a mirror. WIRE now also RETROFITS a 256 B buffer onto
legacy buffer-less bridges (refused under the heap floor) – the silent
black-hole class is gone. METERS: every port row shows its own \u2195 B/s (1 s window, internal traffic included) and THROUGHPUT gains
Switched – everything the fanout carries, boundary AND internal; TX/RX
only ever saw the boundary. OPERATOR: a Reboot hub button
(App.safe_reboot, deferred ack, confirm guard), a raw cmd > mini-console
in SYSTEM (every command now has a handle; replies land in the log), a
t:"pdel" notice on port deletion (the browser closes the matching COM
link – zombie links are extinct), and PORT ADOPT <id> + a Re-link
button per link (re-attach an open COM to a slot-less bridge after a hub
reboot instead of delete-and-recreate).