1. tcp_write infinite loop: When the output byte queue enters an error
state (buffer full), tcp_write would loop forever because
byte_queue_write_buf returns {NULL,0} without advancing. Break out
of the loop when the write buffer is empty.
2. Proxy ignores async return values: toastyfs_async_get/put/delete
can fail (return -1) but the proxy unconditionally set the operation
to STARTED, causing it to wait forever for a result that never comes.
Now checks return values and responds with 500 on failure.
3. Use-after-free on closed connections: When a client disconnects while
its request is queued as PENDING, the HTTP_Conn is freed but the proxy
still holds stale pointers. Added http_response_builder_is_valid() to
detect and discard operations for dead connections.
4. Wrong HTTP status codes: The proxy returned 500 for all errors. Now
returns 507 (Insufficient Storage) for FULL, 404 for NOT_FOUND, and
503 (Service Unavailable) when the operation queue is full.
5. TCP connection leak (root cause of proxy hang): When a peer closes a
connection, process_conn_events closes the socket and sets HUP, but
tcp_conn_free_maybe immediately frees the TCP_Conn before
tcp_next_event can deliver the HUP event. The HTTP server never learns
about the disconnect, so HTTP_Conn entries accumulate until max_conns
is reached and the proxy stops accepting new connections. Fixed by
only calling tcp_conn_free_maybe when the CLOSED flag is already set
(meaning the user called tcp_close), preserving the connection for
HUP event delivery otherwise.
Also adds wrk-based stress test suite (stress-test.sh + lua scripts).
https://claude.ai/code/session_01EHoWWnVgyCFxah5WaNV4kS
Two bugs prevented the cluster from working after a stop/start cycle:
1. tcp_listen_tcp() and tcp_listen_tls() had reuse_addr set to false,
causing bind() to fail with EADDRINUSE when ports are in TIME_WAIT
state after a restart. Set SO_REUSEADDR to true.
2. The vsr_boot_marker file (created in CWD on first boot) was never
cleaned up on cluster stop. On restart, all nodes would enter
STATUS_RECOVERY, but since RECOVERY messages are only processed by
nodes in STATUS_NORMAL, the entire cluster would deadlock with no
node able to recover. Remove the marker in cluster.sh stop.
https://claude.ai/code/session_01U72omHJrk95xFwm22Rkqgt
- Use closesocket() instead of close() for sockets on Windows (lib/tcp.c)
- Call WSAStartup in tcp_init so outbound connections work (lib/tcp.c)
- Check WSAGetLastError() instead of errno for recv/send on Windows (lib/tcp.c)
- Use WSAEWOULDBLOCK instead of EINPROGRESS for non-blocking connect (lib/tcp.c)
- Add NULL guard in tcp_write for stale connection handles (lib/tcp.c)
- Fix ConnMetadata/TCP_Conn index mismatch in get_next_message: use
event.handle.idx directly instead of searching for a free metadata
slot, which could diverge when stale entries remain from failed
outbound connections (lib/message.c)
- Initialize num_senders to 0 and skip unused metadata in
find_conn_by_target (lib/message.c)
- Clear ready flag in http_server_next_request to prevent infinite
loop (lib/http_server.c)
- Add CRT invalid parameter handler for the HTTP proxy (src/main.c)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claudeandcozis