Four bugs caused data mismatches between PUT and GET operations:
1. chunk_store_exists() used file_open (O_CREAT) to check if a chunk
file exists, which created empty files as a side effect. Servers
would then read from these empty files and return uninitialized
heap memory as chunk data. Fixed by using file_exists (access())
instead.
2. Server's process_fetch_chunk checked `ret < 0` after
chunk_store_read, missing the EOF case (ret == 0) from empty
files. Changed to `ret <= 0`.
3. begin_transfers() == 0 was used as a completion check in both
STEP_FETCH_CHUNK and STEP_STORE_CHUNK, but returning 0 only means
"no new transfers started", not "all done". With concurrent chunk
transfers, this caused premature completion, leaving in-flight
transfers to hit UNREACHABLE. Fixed by using
all_chunk_transfers_completed() instead.
4. choose_store_locations_for_chunk() returned [0, 1] for ALL chunks
regardless of index, while GET fetches chunk i from servers
(i+j) % num_servers. This made some chunks unreachable. Fixed by
using (chunk_idx + j) % num_servers for store locations.
Also added SO_REUSEADDR to the server listen socket so restarting
the cluster doesn't fail with "Couldn't setup TCP listener", and
added a note on mock_access about the simulation mismatch.
https://claude.ai/code/session_01JniNxxDP4NbsDXNGmGNG7H
Introduce a Makefile that builds libtoastyfs.a and libtoastyfs.so from the
client-side sources (client, tcp, byte_queue, message, basic) so external
programs can link against the ToastyFS client API in include/toastyfs.h.
Add cluster.sh, a convenience script that manages a local 3-node cluster
by spawning toastyfs server processes on ports 8081-8083. Sub-commands:
up, down, status, logs, and "run <source.c>" which builds the library,
compiles the source against it, starts the cluster, runs the binary, and
tears everything down.
Add examples/example.c demonstrating synchronous PUT, GET, DELETE, and a
NOT_FOUND verification.
Fix a stack overflow in the MAIN_SERVER entry point: ServerState is ~40 MB
(MetaStore holds 4096 ObjectMeta entries) and was declared on the stack.
Heap-allocate it instead.
https://claude.ai/code/session_019DVhmc25jfzcHnmdNxzib2
Introduce a Makefile that builds libtoastyfs.a and libtoastyfs.so from the
client-side sources (client, tcp, byte_queue, message, basic) so external
programs can link against the ToastyFS client API in include/toastyfs.h.
Add a Dockerfile and docker-compose.yml that spin up a 3-node ToastyFS
server cluster on a private bridge network with ports 8081-8083 exposed
on the host. Include cluster.sh as a convenience wrapper (up/down/status/
logs/run) — the "run" sub-command builds the library, compiles a user-
supplied C source against it, starts the cluster, and executes the binary.
Add examples/example.c demonstrating synchronous PUT, GET, DELETE, and a
NOT_FOUND verification.
https://claude.ai/code/session_019DVhmc25jfzcHnmdNxzib2
- Move the CEIL macro from config.h to basic.h alongside MIN/MAX, and
use it in deadline_to_timeout for the rounding-up division
- Remove the heartbeat reset in complete_view_change_and_become_primary
since it was not needed — heartbeat is already set when the node
enters STATUS_CHANGE_VIEW, which is close enough in time
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
The client library had several issues preventing operations from completing
in the simulation:
- toastyfs_register_events discarded the computed timeout instead of
returning it to callers, so clients never woke up on schedule
- random_client_tick ignored the timeout parameter entirely
- deadline_to_timeout truncated sub-millisecond values to 0 instead of
rounding up, causing busy-wait loops
- The server didn't reset the heartbeat timer after completing a view
change, so followers would immediately timeout again
- The client's DELETE handler didn't handle META_RESULT_NOT_FOUND,
causing an assertion failure when deleting non-existent keys
- Most critically, messages sent before TCP connections were established
were silently dropped with no retry mechanism — added timeout-based
retry logic in toastyfs_process_events
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
Add logging infrastructure to client.c mirroring the server's node_log
pattern, with trace lines for: init, async put/get/delete, begin
transfers, replay requests, all received message types (store ack, fetch
chunk response, redirect, reply, get blob response), disconnects, and
unexpected messages.
Add result/operation trace lines to random_client.c and fix a bug where
the random client never started its first operation (the condition
`result.type != TOASTYFS_RESULT_VOID` was always false on the first
tick since no operation was pending).
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
- Add #warning to remind about replacing compute_chunk_hash with proper SHA256
- Define MAX_TRANSFERS and CEIL macros in config.h and use them
- Add TODO noting server selection formula is temporary
- Add NOTE about sticky error pattern in async functions
- Add TODO for INVALID_TIME error handling at all get_current_time call sites
- Add TODO about determining timeout based on pending operation status
https://claude.ai/code/session_01TkNavR4qyZcTuXMPmrtWUJ
- Add REPLICATION_FACTOR and CHUNK_SIZE constants to config.h
- Add proper error codes (REJECTED, FULL, NOT_FOUND, TRANSFER_FAILED, etc.) to toastyfs.h
- Implement compute_chunk_hash() for deterministic chunk hashing
- Implement replay_request() to resend operations after leader redirect
- Implement send_message_to_server_ex() for sending messages with trailing data
- Implement choose_store_locations_for_chunk() to pick REPLICATION_FACTOR servers
- Fix begin_transfers() to handle both StoreChunk (PUT) and FetchChunk (GET)
- Complete toastyfs_async_put(): chunk splitting, hashing, data copy, metadata setup
- Complete toastyfs_async_get()/delete(): key setup, step/time initialization
- Fix get_result(): use actual transfer sizes instead of fixed chunk_size, free transfer data
- Replace all xxx placeholders and TOASTYFS_ERROR_XXX with proper values
- Add chunk_sizes[] tracking and put_data lifetime management
- Implement random_client.c: rng(), choose_random_oper(), proper API call arguments
https://claude.ai/code/session_01TkNavR4qyZcTuXMPmrtWUJ
The node_log macro used a 256-byte buffer for formatting log details,
but bucket names (64B) and keys (512B) can together require up to ~606
bytes, causing gcc to emit format-truncation warnings. Increase the
buffer to 1024 bytes to accommodate the worst case.
https://claude.ai/code/session_01HXKM6UjE3G4A29zTc7gFxo
- Rename phase_time to step_time for consistency with Step enum
- Unify blob_size and file_size into single blob_size field
- Merge toastyfs_alloc into toastyfs_init, returning malloced ToastyFS*
- Replace (MessageHeader*)&msg casts with &msg.base
- Replace !resp.size with explicit resp.size == 0
https://claude.ai/code/session_018MRQTdAZoBCXJZgdP4U6V6
1. Fix client not reading all server addresses in READ response
The metadata server sends address info for ALL chunk servers holding
a chunk, but the client only read info for ONE server. This caused
message parsing to become misaligned for subsequent chunks.
Added a loop to iterate through all num_servers and read each
server's address list (using the first one, skipping the rest).
2. Fix tag value conflict between metadata and chunk server responses
TAG_RETRIEVE_METADATA_FOR_READ was 1, which conflicted with chunk
download range indices (0, 1, 2, 3...). When the second chunk
download response came with range_idx=1, it was incorrectly treated
as a metadata response.
Changed TAG values from positive to negative integers since range
indices are always non-negative.
Also updated the test to properly exit on success instead of assert(0).
https://claude.ai/code/session_01QYDQCNiUj1cjrsS6qRgkvd
The upload tag is computed as TAG_UPLOAD_CHUNK_MIN + upload_index, and the
assertion tag <= TAG_UPLOAD_CHUNK_MAX was failing when the number of uploads
exceeded 1000. This can happen with small chunk sizes (e.g., 1 byte) and
large writes, where each chunk needs multiple upload operations for
replication.
Increase TAG_UPLOAD_CHUNK_MAX from 2000 to 100000 to accommodate up to 99000
concurrent upload operations per write.
When responding to a full chunk download request (full == 1), the code
was writing uninitialized target_len to the response message instead of
the actual data length. This caused chunk-to-chunk downloads to fail,
preventing proper replication and blocking coverage of download-related
code paths.
Two changes to improve code coverage in deterministic simulation testing:
1. Add 3 chunk servers (was 1) to match the replication factor
- This enables testing of chunk replication and chunk-to-chunk
server communication code paths in chunk_server.c
- Covers: process_chunk_server_download_*, start_download,
download_targets_* functions
2. Fix Quakey bug where events targeting a closed descriptor caused
assertion failures
- Added remove_events_targeting_desc() to clean up DISCONNECT and
DATA events when a descriptor is freed
- Modified desc_free() to accept Sim pointer and call cleanup
- This was exposed by running multiple chunk servers which create
more connection activity
Three related issues were causing the simulation to get stuck:
1. Network events (connect, disconnect, send_data) were scheduled with
relative times instead of absolute times, causing events to be
scheduled in the past once simulation time exceeded 10-100ms.
2. When DATA events couldn't be fully consumed (receiver buffer full),
they stayed at their original time, preventing time from advancing.
Now they are rescheduled to current_time + 10ms.
3. When poll_timeout was 0, WAKEUP events were created at current_time,
causing an infinite loop where time never advanced. Now timeout=0
sets timedout immediately without creating an event.
cozis
Claude