kwerty.space // systems & network engineering log

Tuning epoll Event Loops and SO_REUSEPORT Worker Affinity Under Burst Loads

When running multi-worker HTTP reverse proxies on modest VM instances with one to four virtual cores, a single shared listening socket protected by an accept lock often causes uneven wake-up distribution. Enabling SO_REUSEPORT lets the Linux kernel hash incoming five-tuples directly to per-worker listening sockets, eliminating cross-core lock contention during sudden connection spikes.

In our synthetic replay tests (4,000 new TLS 1.3 handshakes per second over 30 seconds), edge-triggered epoll_wait batching combined with EPOLLEXCLUSIVE reduced p99 accept latency from 18.4 ms down to 4.1 ms. Careful sizing of net.core.somaxconn and the SYN backlog ensures that temporary scheduler stalls inside userspace never spill into silent SYN drops.

# /etc/sysctl.d/99-socket-tuning.conf
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 8192
net.ipv4.tcp_fastopen = 3

# Per-worker epoll accept loop benchmark (2 workers, 10k concurrent sockets)
# Baseline shared socket:  p50 = 1.9ms | p95 = 7.8ms | p99 = 18.4ms | drops = 14
# SO_REUSEPORT + reuseport_cbpf: p50 = 1.2ms | p95 = 2.6ms | p99 = 4.1ms | drops = 0

One caveat worth remembering on container hosts or virtualized NICs: if hardware RSS queues are limited to a single IRQ vector, pinning nginx worker processes without inspecting /proc/interrupts can cause CPU starvation on core 0. Always balance IRQ affinity before locking worker CPUs in production.

#linux #epoll #kernel

HTTP/2 Multiplexing, Flow-Control Windows, and Head-of-Line Effects in Reverse Proxies

HTTP/2 resolves application-layer head-of-line blocking across concurrent requests on the same origin, yet default flow-control window parameters frequently become the real bottleneck across high-BDP WAN links. With a standard 64 KiB initial window size on a 90 ms round-trip path, a single multiplexed stream stalls waiting for WINDOW_UPDATE frames long before saturating a 100 Mbps link.

Expanding downstream stream buffers while simultaneously disabling unnecessary disk buffering for long-lived SSE or chunked streams keeps memory predictable. On Nginx origins sitting behind an upstream TLS terminator, explicitly emitting X-Accel-Buffering: no guarantees flush parity even when intermediate relays inspect headers.

location /stream-api/ {
    proxy_pass http://upstream_pool;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_request_buffering off;
    proxy_read_timeout 3600s;
    add_header X-Accel-Buffering "no" always;
}

# Measured throughput (92 ms RTT, single HTTP/2 connection, 256 KiB vs 2 MiB window):
#  64 KiB window ->  5.4 Mbps peak | avg TTFF 195 ms
# 512 KiB window -> 41.2 Mbps peak | avg TTFF 189 ms
#   2 MiB window -> 94.8 Mbps peak | avg TTFF 191 ms

Note that increasing HTTP/2 windows indiscriminately on memory-constrained nodes (1 GiB RAM) exposes workers to slow-reader memory pressure if hundreds of idle streams advertise large receive windows without backpressure. Pairing keepalive connection pools with strict per-worker connection limits provides the cleanest safety boundary.

#http2 #linux

NVMe Queue Depth vs. Tail Latency: none vs. kyber I/O Schedulers on Cloud Volumes

Modern cloud hypervisors back virtual NVMe namespaces either with local PCIe passthrough drives or distributed replicated block storage. In both deployments, Linux defaults can leave multi-queue scheduling suboptimally tuned when mixed workloads combine sequential access-log appends with random SQLite metadata writes.

We ran fio 4 KiB random read/write mixes (70/30) across queue depths 1 through 32 on an Ubuntu 24.04 instance to compare none against kyber and mq-deadline. While sequential throughput barely shifted, write-induced read tail latency varied dramatically once queue depth crossed 16.

fio --name=randrw --rw=randrw --rwmixread=70 --bs=4k --iodepth=16 \
    --ioengine=io_uring --direct=1 --runtime=60 --group_reporting

# Scheduler comparison (4 KiB randrw 70/30, iodepth=16):
# scheduler=none        | read IOPS 18.4k | p99 read 2.84 ms | write p99 4.10 ms
# scheduler=mq-deadline | read IOPS 17.9k | p99 read 3.45 ms | write p99 3.72 ms
# scheduler=kyber       | read IOPS 18.2k | p99 read 1.62 ms | write p99 4.95 ms

For read-latency-sensitive services running alongside periodic SQLite checkpoints or systemd journal flushes, kyber consistently protected read tail latency. Conversely, trimming noisy per-request access logs off slow virtual disks saved more I/O budget than any scheduler swap alone.

#storage #kernel #linux

TLS 1.3 Session Resumption, Ticket Keys, and Short-Lived Certificates in Practice

Every full TLS 1.3 handshake adds one complete network round-trip before application bytes move, plus asymmetric signature generation on the edge node. Enabling stateless session tickets alongside shared session caches lets returning clients resume via PSK handshakes cleanly while preserving forward secrecy across rotated ticket keys.

Short-lived 6-day IP certificates issued via automated ACME webroot challenges introduce an interesting operational trade-off: OCSP stapling is often omitted or unnecessary when validity windows are measured in days rather than months, reducing external CA lookups and certificate chain payload overhead inside the ServerHello flight.

ssl_protocols TLSv1.2 TLSv1.3;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 1d;
ssl_buffer_size 4k;

# Handshake bytes & latency over 74 ms RTT link:
# Full TLS 1.3 ECDSA P-256 handshake: ~2,410 bytes total | ~154 ms to first app byte
# Resumed TLS 1.3 PSK handshake:       ~480 bytes total  | ~78 ms to first app byte

Reducing ssl_buffer_size from Nginx's 16 KiB default down to 4 KiB also noticeably trims first-byte latency on small JSON responses and health probes, because the server flushes encrypted TLS records immediately instead of coalescing extra frames.

#tls13 #http2

Anycast BGP ECMP Path Hashing and Measuring RTT Variance Across Tier-1 Transits

Public Anycast edge nodes distribute inbound traffic across points of presence via BGP announcements and Equal-Cost Multi-Path (ECMP) routing. From an end-user perspective, slight changes in source port or IPv4 vs. IPv6 address families can hash traffic onto completely different transit carriers—one exhibiting zero packet loss and another suffering evening congestion bursts.

When debugging intermittent stalls that only affect specific client subnets or address families, automated periodic HTTP 204 latency sampling reveals path asymmetry much faster than raw ICMP pings, because middleboxes frequently deprioritize or rate-limit ICMP echo traffic differently from TCP port 443 flows.

# Example dual-stack probe comparison across Anycast edge POPs
$ curl -4 -s -o /dev/null -w "v4_connect=%{time_connect}s v4_tls=%{time_appconnect}s\n" https://edge-pop.example/
v4_connect=0.028s v4_tls=0.144s

$ curl -6 -s -o /dev/null -w "v6_connect=%{time_connect}s v6_tls=%{time_appconnect}s\n" https://edge-pop.example/
v6_connect=0.092s v6_tls=0.197s

Keeping active client health checks decoupled across independent POPs allows applications to shift away from degraded transits within seconds rather than waiting on upstream BGP convergence timers.

#bgp #kernel #linux