Four bugs caused data mismatches between PUT and GET operations:
1. chunk_store_exists() used file_open (O_CREAT) to check if a chunk
file exists, which created empty files as a side effect. Servers
would then read from these empty files and return uninitialized
heap memory as chunk data. Fixed by using file_exists (access())
instead.
2. Server's process_fetch_chunk checked `ret < 0` after
chunk_store_read, missing the EOF case (ret == 0) from empty
files. Changed to `ret <= 0`.
3. begin_transfers() == 0 was used as a completion check in both
STEP_FETCH_CHUNK and STEP_STORE_CHUNK, but returning 0 only means
"no new transfers started", not "all done". With concurrent chunk
transfers, this caused premature completion, leaving in-flight
transfers to hit UNREACHABLE. Fixed by using
all_chunk_transfers_completed() instead.
4. choose_store_locations_for_chunk() returned [0, 1] for ALL chunks
regardless of index, while GET fetches chunk i from servers
(i+j) % num_servers. This made some chunks unreachable. Fixed by
using (chunk_idx + j) % num_servers for store locations.
Also added SO_REUSEADDR to the server listen socket so restarting
the cluster doesn't fail with "Couldn't setup TCP listener", and
added a note on mock_access about the simulation mismatch.
https://claude.ai/code/session_01JniNxxDP4NbsDXNGmGNG7H
Introduce a Makefile that builds libtoastyfs.a and libtoastyfs.so from the
client-side sources (client, tcp, byte_queue, message, basic) so external
programs can link against the ToastyFS client API in include/toastyfs.h.
Add cluster.sh, a convenience script that manages a local 3-node cluster
by spawning toastyfs server processes on ports 8081-8083. Sub-commands:
up, down, status, logs, and "run <source.c>" which builds the library,
compiles the source against it, starts the cluster, runs the binary, and
tears everything down.
Add examples/example.c demonstrating synchronous PUT, GET, DELETE, and a
NOT_FOUND verification.
Fix a stack overflow in the MAIN_SERVER entry point: ServerState is ~40 MB
(MetaStore holds 4096 ObjectMeta entries) and was declared on the stack.
Heap-allocate it instead.
https://claude.ai/code/session_019DVhmc25jfzcHnmdNxzib2
Introduce a Makefile that builds libtoastyfs.a and libtoastyfs.so from the
client-side sources (client, tcp, byte_queue, message, basic) so external
programs can link against the ToastyFS client API in include/toastyfs.h.
Add a Dockerfile and docker-compose.yml that spin up a 3-node ToastyFS
server cluster on a private bridge network with ports 8081-8083 exposed
on the host. Include cluster.sh as a convenience wrapper (up/down/status/
logs/run) — the "run" sub-command builds the library, compiles a user-
supplied C source against it, starts the cluster, and executes the binary.
Add examples/example.c demonstrating synchronous PUT, GET, DELETE, and a
NOT_FOUND verification.
https://claude.ai/code/session_019DVhmc25jfzcHnmdNxzib2
- Move the CEIL macro from config.h to basic.h alongside MIN/MAX, and
use it in deadline_to_timeout for the rounding-up division
- Remove the heartbeat reset in complete_view_change_and_become_primary
since it was not needed — heartbeat is already set when the node
enters STATUS_CHANGE_VIEW, which is close enough in time
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
The client library had several issues preventing operations from completing
in the simulation:
- toastyfs_register_events discarded the computed timeout instead of
returning it to callers, so clients never woke up on schedule
- random_client_tick ignored the timeout parameter entirely
- deadline_to_timeout truncated sub-millisecond values to 0 instead of
rounding up, causing busy-wait loops
- The server didn't reset the heartbeat timer after completing a view
change, so followers would immediately timeout again
- The client's DELETE handler didn't handle META_RESULT_NOT_FOUND,
causing an assertion failure when deleting non-existent keys
- Most critically, messages sent before TCP connections were established
were silently dropped with no retry mechanism — added timeout-based
retry logic in toastyfs_process_events
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
Add logging infrastructure to client.c mirroring the server's node_log
pattern, with trace lines for: init, async put/get/delete, begin
transfers, replay requests, all received message types (store ack, fetch
chunk response, redirect, reply, get blob response), disconnects, and
unexpected messages.
Add result/operation trace lines to random_client.c and fix a bug where
the random client never started its first operation (the condition
`result.type != TOASTYFS_RESULT_VOID` was always false on the first
tick since no operation was pending).
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
- Add #warning to remind about replacing compute_chunk_hash with proper SHA256
- Define MAX_TRANSFERS and CEIL macros in config.h and use them
- Add TODO noting server selection formula is temporary
- Add NOTE about sticky error pattern in async functions
- Add TODO for INVALID_TIME error handling at all get_current_time call sites
- Add TODO about determining timeout based on pending operation status
https://claude.ai/code/session_01TkNavR4qyZcTuXMPmrtWUJ
- Add REPLICATION_FACTOR and CHUNK_SIZE constants to config.h
- Add proper error codes (REJECTED, FULL, NOT_FOUND, TRANSFER_FAILED, etc.) to toastyfs.h
- Implement compute_chunk_hash() for deterministic chunk hashing
- Implement replay_request() to resend operations after leader redirect
- Implement send_message_to_server_ex() for sending messages with trailing data
- Implement choose_store_locations_for_chunk() to pick REPLICATION_FACTOR servers
- Fix begin_transfers() to handle both StoreChunk (PUT) and FetchChunk (GET)
- Complete toastyfs_async_put(): chunk splitting, hashing, data copy, metadata setup
- Complete toastyfs_async_get()/delete(): key setup, step/time initialization
- Fix get_result(): use actual transfer sizes instead of fixed chunk_size, free transfer data
- Replace all xxx placeholders and TOASTYFS_ERROR_XXX with proper values
- Add chunk_sizes[] tracking and put_data lifetime management
- Implement random_client.c: rng(), choose_random_oper(), proper API call arguments
https://claude.ai/code/session_01TkNavR4qyZcTuXMPmrtWUJ
The node_log macro used a 256-byte buffer for formatting log details,
but bucket names (64B) and keys (512B) can together require up to ~606
bytes, causing gcc to emit format-truncation warnings. Increase the
buffer to 1024 bytes to accommodate the worst case.
https://claude.ai/code/session_01HXKM6UjE3G4A29zTc7gFxo
- Rename phase_time to step_time for consistency with Step enum
- Unify blob_size and file_size into single blob_size field
- Merge toastyfs_alloc into toastyfs_init, returning malloced ToastyFS*
- Replace (MessageHeader*)&msg casts with &msg.base
- Replace !resp.size with explicit resp.size == 0
https://claude.ai/code/session_018MRQTdAZoBCXJZgdP4U6V6
1. Fix client not reading all server addresses in READ response
The metadata server sends address info for ALL chunk servers holding
a chunk, but the client only read info for ONE server. This caused
message parsing to become misaligned for subsequent chunks.
Added a loop to iterate through all num_servers and read each
server's address list (using the first one, skipping the rest).
2. Fix tag value conflict between metadata and chunk server responses
TAG_RETRIEVE_METADATA_FOR_READ was 1, which conflicted with chunk
download range indices (0, 1, 2, 3...). When the second chunk
download response came with range_idx=1, it was incorrectly treated
as a metadata response.
Changed TAG values from positive to negative integers since range
indices are always non-negative.
Also updated the test to properly exit on success instead of assert(0).
https://claude.ai/code/session_01QYDQCNiUj1cjrsS6qRgkvd
The upload tag is computed as TAG_UPLOAD_CHUNK_MIN + upload_index, and the
assertion tag <= TAG_UPLOAD_CHUNK_MAX was failing when the number of uploads
exceeded 1000. This can happen with small chunk sizes (e.g., 1 byte) and
large writes, where each chunk needs multiple upload operations for
replication.
Increase TAG_UPLOAD_CHUNK_MAX from 2000 to 100000 to accommodate up to 99000
concurrent upload operations per write.
When responding to a full chunk download request (full == 1), the code
was writing uninitialized target_len to the response message instead of
the actual data length. This caused chunk-to-chunk downloads to fail,
preventing proper replication and blocking coverage of download-related
code paths.
Two changes to improve code coverage in deterministic simulation testing:
1. Add 3 chunk servers (was 1) to match the replication factor
- This enables testing of chunk replication and chunk-to-chunk
server communication code paths in chunk_server.c
- Covers: process_chunk_server_download_*, start_download,
download_targets_* functions
2. Fix Quakey bug where events targeting a closed descriptor caused
assertion failures
- Added remove_events_targeting_desc() to clean up DISCONNECT and
DATA events when a descriptor is freed
- Modified desc_free() to accept Sim pointer and call cleanup
- This was exposed by running multiple chunk servers which create
more connection activity
Three related issues were causing the simulation to get stuck:
1. Network events (connect, disconnect, send_data) were scheduled with
relative times instead of absolute times, causing events to be
scheduled in the past once simulation time exceeded 10-100ms.
2. When DATA events couldn't be fully consumed (receiver buffer full),
they stayed at their original time, preventing time from advancing.
Now they are rescheduled to current_time + 10ms.
3. When poll_timeout was 0, WAKEUP events were created at current_time,
causing an infinite loop where time never advanced. Now timeout=0
sets timedout immediately without creating an event.
Initialize *gen to NO_GENERATION at the start of file_tree_read so that
error paths (FILETREE_BADPATH, FILETREE_NOENT, FILETREE_ISDIR) have a
well-defined value. Previously, when file_tree_read failed, gen was
uninitialized, causing the assertion in process_client_read to fire.
The assertion checking gen != NO_GENERATION before the error check was
removed since:
1. On error paths, gen is not used
2. On success paths, the assertion at line 482 already validates gen
Simplifies the script to automatically compile, run (60s), and generate
the coverage report in a single invocation. Removes the two-step --report
flag approach and adds quieter output with clear step markers.
Adds build_coverage.sh script that:
- Builds the simulation with GCC branch coverage instrumentation
- Supports --report flag to run simulation and generate lcov HTML report
- Uses 60 second timeout with SIGINT for graceful shutdown
Also adds SIGINT handler to simulation to exit cleanly and flush coverage data.
Claude
cozis