1. tcp_write infinite loop: When the output byte queue enters an error
state (buffer full), tcp_write would loop forever because
byte_queue_write_buf returns {NULL,0} without advancing. Break out
of the loop when the write buffer is empty.
2. Proxy ignores async return values: toastyfs_async_get/put/delete
can fail (return -1) but the proxy unconditionally set the operation
to STARTED, causing it to wait forever for a result that never comes.
Now checks return values and responds with 500 on failure.
3. Use-after-free on closed connections: When a client disconnects while
its request is queued as PENDING, the HTTP_Conn is freed but the proxy
still holds stale pointers. Added http_response_builder_is_valid() to
detect and discard operations for dead connections.
4. Wrong HTTP status codes: The proxy returned 500 for all errors. Now
returns 507 (Insufficient Storage) for FULL, 404 for NOT_FOUND, and
503 (Service Unavailable) when the operation queue is full.
5. TCP connection leak (root cause of proxy hang): When a peer closes a
connection, process_conn_events closes the socket and sets HUP, but
tcp_conn_free_maybe immediately frees the TCP_Conn before
tcp_next_event can deliver the HUP event. The HTTP server never learns
about the disconnect, so HTTP_Conn entries accumulate until max_conns
is reached and the proxy stops accepting new connections. Fixed by
only calling tcp_conn_free_maybe when the CLOSED flag is already set
(meaning the user called tcp_close), preserving the connection for
HUP event delivery otherwise.
Also adds wrk-based stress test suite (stress-test.sh + lua scripts).
https://claude.ai/code/session_01EHoWWnVgyCFxah5WaNV4kS
Two bugs prevented the cluster from working after a stop/start cycle:
1. tcp_listen_tcp() and tcp_listen_tls() had reuse_addr set to false,
causing bind() to fail with EADDRINUSE when ports are in TIME_WAIT
state after a restart. Set SO_REUSEADDR to true.
2. The vsr_boot_marker file (created in CWD on first boot) was never
cleaned up on cluster stop. On restart, all nodes would enter
STATUS_RECOVERY, but since RECOVERY messages are only processed by
nodes in STATUS_NORMAL, the entire cluster would deadlock with no
node able to recover. Remove the marker in cluster.sh stop.
https://claude.ai/code/session_01U72omHJrk95xFwm22Rkqgt
Provides start/stop/status/restart/clean commands to manage a local
ToastyFS cluster. Nodes listen on 127.0.0.1:8001-8003 with the HTTP
proxy on 127.0.0.1:3000. Each node gets its own chunk storage directory
under cluster-data/. Logs and PID files are stored there as well.
https://claude.ai/code/session_01FAL4T35UGYqXexbaxvihRp
- Use closesocket() instead of close() for sockets on Windows (lib/tcp.c)
- Call WSAStartup in tcp_init so outbound connections work (lib/tcp.c)
- Check WSAGetLastError() instead of errno for recv/send on Windows (lib/tcp.c)
- Use WSAEWOULDBLOCK instead of EINPROGRESS for non-blocking connect (lib/tcp.c)
- Add NULL guard in tcp_write for stale connection handles (lib/tcp.c)
- Fix ConnMetadata/TCP_Conn index mismatch in get_next_message: use
event.handle.idx directly instead of searching for a free metadata
slot, which could diverge when stale entries remain from failed
outbound connections (lib/message.c)
- Initialize num_senders to 0 and skip unused metadata in
find_conn_by_target (lib/message.c)
- Clear ready flag in http_server_next_request to prevent infinite
loop (lib/http_server.c)
- Add CRT invalid parameter handler for the HTTP proxy (src/main.c)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
file_truncate had a stub `return -1` on Windows, causing all
chunk_store_write calls to fail (every PUT returned TRANSFER_FAILED).
Implement it using SetFilePointer + SetEndOfFile.
Add mock_SetEndOfFile and mock_GetFileAttributesA to Quakey so the
simulation can exercise the Windows code paths against the mock
filesystem. Without these, SetEndOfFile was unmocked (link error) and
GetFileAttributesA called the real Win32 API on mock paths, making
chunk_store_exists always return false.
Verified: seeds 1-100 pass with 2000s simulation time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The COMMIT_PUT message was built with request_id N, then request_id
was incremented to N+1 immediately after sending. When the server
replied with request_id N, the client compared it against N+1,
silently discarded the reply, and waited for the 5-second timeout.
The timeout retry (replay_request) then resent with N+1, which
matched — so every PUT paid an unnecessary 5-second penalty.
Move the request_id++ before building the CommitPutMessage, matching
the pattern already used by toastyfs_async_delete.
https://claude.ai/code/session_01EG3WZSH3ZeTdAbb5T42r2w
When no TCP connection exists, send_message_to_server() called
tcp_connect() with a NULL output parameter and returned without
sending the message. The message was silently dropped, and the only
recovery was the 5-second PRIMARY_DEATH_TIMEOUT_SEC retry — causing
the client to hang on every cold-connection operation.
tcp_connect() already supports returning the output ByteQueue so
callers can queue data to be flushed once the connection completes.
Use that mechanism: get the output queue from tcp_connect() and
write the message into it immediately, so it is sent as soon as
the TCP handshake finishes.
https://claude.ai/code/session_01EG3WZSH3ZeTdAbb5T42r2w
Four bugs caused data mismatches between PUT and GET operations:
1. chunk_store_exists() used file_open (O_CREAT) to check if a chunk
file exists, which created empty files as a side effect. Servers
would then read from these empty files and return uninitialized
heap memory as chunk data. Fixed by using file_exists (access())
instead.
2. Server's process_fetch_chunk checked `ret < 0` after
chunk_store_read, missing the EOF case (ret == 0) from empty
files. Changed to `ret <= 0`.
3. begin_transfers() == 0 was used as a completion check in both
STEP_FETCH_CHUNK and STEP_STORE_CHUNK, but returning 0 only means
"no new transfers started", not "all done". With concurrent chunk
transfers, this caused premature completion, leaving in-flight
transfers to hit UNREACHABLE. Fixed by using
all_chunk_transfers_completed() instead.
4. choose_store_locations_for_chunk() returned [0, 1] for ALL chunks
regardless of index, while GET fetches chunk i from servers
(i+j) % num_servers. This made some chunks unreachable. Fixed by
using (chunk_idx + j) % num_servers for store locations.
Also added SO_REUSEADDR to the server listen socket so restarting
the cluster doesn't fail with "Couldn't setup TCP listener", and
added a note on mock_access about the simulation mismatch.
https://claude.ai/code/session_01JniNxxDP4NbsDXNGmGNG7H
Introduce a Makefile that builds libtoastyfs.a and libtoastyfs.so from the
client-side sources (client, tcp, byte_queue, message, basic) so external
programs can link against the ToastyFS client API in include/toastyfs.h.
Add cluster.sh, a convenience script that manages a local 3-node cluster
by spawning toastyfs server processes on ports 8081-8083. Sub-commands:
up, down, status, logs, and "run <source.c>" which builds the library,
compiles the source against it, starts the cluster, runs the binary, and
tears everything down.
Add examples/example.c demonstrating synchronous PUT, GET, DELETE, and a
NOT_FOUND verification.
Fix a stack overflow in the MAIN_SERVER entry point: ServerState is ~40 MB
(MetaStore holds 4096 ObjectMeta entries) and was declared on the stack.
Heap-allocate it instead.
https://claude.ai/code/session_019DVhmc25jfzcHnmdNxzib2
Introduce a Makefile that builds libtoastyfs.a and libtoastyfs.so from the
client-side sources (client, tcp, byte_queue, message, basic) so external
programs can link against the ToastyFS client API in include/toastyfs.h.
Add a Dockerfile and docker-compose.yml that spin up a 3-node ToastyFS
server cluster on a private bridge network with ports 8081-8083 exposed
on the host. Include cluster.sh as a convenience wrapper (up/down/status/
logs/run) — the "run" sub-command builds the library, compiles a user-
supplied C source against it, starts the cluster, and executes the binary.
Add examples/example.c demonstrating synchronous PUT, GET, DELETE, and a
NOT_FOUND verification.
https://claude.ai/code/session_019DVhmc25jfzcHnmdNxzib2
- Move the CEIL macro from config.h to basic.h alongside MIN/MAX, and
use it in deadline_to_timeout for the rounding-up division
- Remove the heartbeat reset in complete_view_change_and_become_primary
since it was not needed — heartbeat is already set when the node
enters STATUS_CHANGE_VIEW, which is close enough in time
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
The client library had several issues preventing operations from completing
in the simulation:
- toastyfs_register_events discarded the computed timeout instead of
returning it to callers, so clients never woke up on schedule
- random_client_tick ignored the timeout parameter entirely
- deadline_to_timeout truncated sub-millisecond values to 0 instead of
rounding up, causing busy-wait loops
- The server didn't reset the heartbeat timer after completing a view
change, so followers would immediately timeout again
- The client's DELETE handler didn't handle META_RESULT_NOT_FOUND,
causing an assertion failure when deleting non-existent keys
- Most critically, messages sent before TCP connections were established
were silently dropped with no retry mechanism — added timeout-based
retry logic in toastyfs_process_events
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
Add logging infrastructure to client.c mirroring the server's node_log
pattern, with trace lines for: init, async put/get/delete, begin
transfers, replay requests, all received message types (store ack, fetch
chunk response, redirect, reply, get blob response), disconnects, and
unexpected messages.
Add result/operation trace lines to random_client.c and fix a bug where
the random client never started its first operation (the condition
`result.type != TOASTYFS_RESULT_VOID` was always false on the first
tick since no operation was pending).
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
- Add #warning to remind about replacing compute_chunk_hash with proper SHA256
- Define MAX_TRANSFERS and CEIL macros in config.h and use them
- Add TODO noting server selection formula is temporary
- Add NOTE about sticky error pattern in async functions
- Add TODO for INVALID_TIME error handling at all get_current_time call sites
- Add TODO about determining timeout based on pending operation status
https://claude.ai/code/session_01TkNavR4qyZcTuXMPmrtWUJ
- Add REPLICATION_FACTOR and CHUNK_SIZE constants to config.h
- Add proper error codes (REJECTED, FULL, NOT_FOUND, TRANSFER_FAILED, etc.) to toastyfs.h
- Implement compute_chunk_hash() for deterministic chunk hashing
- Implement replay_request() to resend operations after leader redirect
- Implement send_message_to_server_ex() for sending messages with trailing data
- Implement choose_store_locations_for_chunk() to pick REPLICATION_FACTOR servers
- Fix begin_transfers() to handle both StoreChunk (PUT) and FetchChunk (GET)
- Complete toastyfs_async_put(): chunk splitting, hashing, data copy, metadata setup
- Complete toastyfs_async_get()/delete(): key setup, step/time initialization
- Fix get_result(): use actual transfer sizes instead of fixed chunk_size, free transfer data
- Replace all xxx placeholders and TOASTYFS_ERROR_XXX with proper values
- Add chunk_sizes[] tracking and put_data lifetime management
- Implement random_client.c: rng(), choose_random_oper(), proper API call arguments
https://claude.ai/code/session_01TkNavR4qyZcTuXMPmrtWUJ
cozis
Claudeandcozis