The COMMIT_PUT message was built with request_id N, then request_id
was incremented to N+1 immediately after sending. When the server
replied with request_id N, the client compared it against N+1,
silently discarded the reply, and waited for the 5-second timeout.
The timeout retry (replay_request) then resent with N+1, which
matched — so every PUT paid an unnecessary 5-second penalty.
Move the request_id++ before building the CommitPutMessage, matching
the pattern already used by toastyfs_async_delete.
https://claude.ai/code/session_01EG3WZSH3ZeTdAbb5T42r2w
When no TCP connection exists, send_message_to_server() called
tcp_connect() with a NULL output parameter and returned without
sending the message. The message was silently dropped, and the only
recovery was the 5-second PRIMARY_DEATH_TIMEOUT_SEC retry — causing
the client to hang on every cold-connection operation.
tcp_connect() already supports returning the output ByteQueue so
callers can queue data to be flushed once the connection completes.
Use that mechanism: get the output queue from tcp_connect() and
write the message into it immediately, so it is sent as soon as
the TCP handshake finishes.
https://claude.ai/code/session_01EG3WZSH3ZeTdAbb5T42r2w
Four bugs caused data mismatches between PUT and GET operations:
1. chunk_store_exists() used file_open (O_CREAT) to check if a chunk
file exists, which created empty files as a side effect. Servers
would then read from these empty files and return uninitialized
heap memory as chunk data. Fixed by using file_exists (access())
instead.
2. Server's process_fetch_chunk checked `ret < 0` after
chunk_store_read, missing the EOF case (ret == 0) from empty
files. Changed to `ret <= 0`.
3. begin_transfers() == 0 was used as a completion check in both
STEP_FETCH_CHUNK and STEP_STORE_CHUNK, but returning 0 only means
"no new transfers started", not "all done". With concurrent chunk
transfers, this caused premature completion, leaving in-flight
transfers to hit UNREACHABLE. Fixed by using
all_chunk_transfers_completed() instead.
4. choose_store_locations_for_chunk() returned [0, 1] for ALL chunks
regardless of index, while GET fetches chunk i from servers
(i+j) % num_servers. This made some chunks unreachable. Fixed by
using (chunk_idx + j) % num_servers for store locations.
Also added SO_REUSEADDR to the server listen socket so restarting
the cluster doesn't fail with "Couldn't setup TCP listener", and
added a note on mock_access about the simulation mismatch.
https://claude.ai/code/session_01JniNxxDP4NbsDXNGmGNG7H
Introduce a Makefile that builds libtoastyfs.a and libtoastyfs.so from the
client-side sources (client, tcp, byte_queue, message, basic) so external
programs can link against the ToastyFS client API in include/toastyfs.h.
Add cluster.sh, a convenience script that manages a local 3-node cluster
by spawning toastyfs server processes on ports 8081-8083. Sub-commands:
up, down, status, logs, and "run <source.c>" which builds the library,
compiles the source against it, starts the cluster, runs the binary, and
tears everything down.
Add examples/example.c demonstrating synchronous PUT, GET, DELETE, and a
NOT_FOUND verification.
Fix a stack overflow in the MAIN_SERVER entry point: ServerState is ~40 MB
(MetaStore holds 4096 ObjectMeta entries) and was declared on the stack.
Heap-allocate it instead.
https://claude.ai/code/session_019DVhmc25jfzcHnmdNxzib2
- Move the CEIL macro from config.h to basic.h alongside MIN/MAX, and
use it in deadline_to_timeout for the rounding-up division
- Remove the heartbeat reset in complete_view_change_and_become_primary
since it was not needed — heartbeat is already set when the node
enters STATUS_CHANGE_VIEW, which is close enough in time
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
The client library had several issues preventing operations from completing
in the simulation:
- toastyfs_register_events discarded the computed timeout instead of
returning it to callers, so clients never woke up on schedule
- random_client_tick ignored the timeout parameter entirely
- deadline_to_timeout truncated sub-millisecond values to 0 instead of
rounding up, causing busy-wait loops
- The server didn't reset the heartbeat timer after completing a view
change, so followers would immediately timeout again
- The client's DELETE handler didn't handle META_RESULT_NOT_FOUND,
causing an assertion failure when deleting non-existent keys
- Most critically, messages sent before TCP connections were established
were silently dropped with no retry mechanism — added timeout-based
retry logic in toastyfs_process_events
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
Add logging infrastructure to client.c mirroring the server's node_log
pattern, with trace lines for: init, async put/get/delete, begin
transfers, replay requests, all received message types (store ack, fetch
chunk response, redirect, reply, get blob response), disconnects, and
unexpected messages.
Add result/operation trace lines to random_client.c and fix a bug where
the random client never started its first operation (the condition
`result.type != TOASTYFS_RESULT_VOID` was always false on the first
tick since no operation was pending).
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
- Add #warning to remind about replacing compute_chunk_hash with proper SHA256
- Define MAX_TRANSFERS and CEIL macros in config.h and use them
- Add TODO noting server selection formula is temporary
- Add NOTE about sticky error pattern in async functions
- Add TODO for INVALID_TIME error handling at all get_current_time call sites
- Add TODO about determining timeout based on pending operation status
https://claude.ai/code/session_01TkNavR4qyZcTuXMPmrtWUJ
- Add REPLICATION_FACTOR and CHUNK_SIZE constants to config.h
- Add proper error codes (REJECTED, FULL, NOT_FOUND, TRANSFER_FAILED, etc.) to toastyfs.h
- Implement compute_chunk_hash() for deterministic chunk hashing
- Implement replay_request() to resend operations after leader redirect
- Implement send_message_to_server_ex() for sending messages with trailing data
- Implement choose_store_locations_for_chunk() to pick REPLICATION_FACTOR servers
- Fix begin_transfers() to handle both StoreChunk (PUT) and FetchChunk (GET)
- Complete toastyfs_async_put(): chunk splitting, hashing, data copy, metadata setup
- Complete toastyfs_async_get()/delete(): key setup, step/time initialization
- Fix get_result(): use actual transfer sizes instead of fixed chunk_size, free transfer data
- Replace all xxx placeholders and TOASTYFS_ERROR_XXX with proper values
- Add chunk_sizes[] tracking and put_data lifetime management
- Implement random_client.c: rng(), choose_random_oper(), proper API call arguments
https://claude.ai/code/session_01TkNavR4qyZcTuXMPmrtWUJ
The node_log macro used a 256-byte buffer for formatting log details,
but bucket names (64B) and keys (512B) can together require up to ~606
bytes, causing gcc to emit format-truncation warnings. Increase the
buffer to 1024 bytes to accommodate the worst case.
https://claude.ai/code/session_01HXKM6UjE3G4A29zTc7gFxo
- Rename phase_time to step_time for consistency with Step enum
- Unify blob_size and file_size into single blob_size field
- Merge toastyfs_alloc into toastyfs_init, returning malloced ToastyFS*
- Replace (MessageHeader*)&msg casts with &msg.base
- Replace !resp.size with explicit resp.size == 0
https://claude.ai/code/session_018MRQTdAZoBCXJZgdP4U6V6
1. Fix client not reading all server addresses in READ response
The metadata server sends address info for ALL chunk servers holding
a chunk, but the client only read info for ONE server. This caused
message parsing to become misaligned for subsequent chunks.
Added a loop to iterate through all num_servers and read each
server's address list (using the first one, skipping the rest).
2. Fix tag value conflict between metadata and chunk server responses
TAG_RETRIEVE_METADATA_FOR_READ was 1, which conflicted with chunk
download range indices (0, 1, 2, 3...). When the second chunk
download response came with range_idx=1, it was incorrectly treated
as a metadata response.
Changed TAG values from positive to negative integers since range
indices are always non-negative.
Also updated the test to properly exit on success instead of assert(0).
https://claude.ai/code/session_01QYDQCNiUj1cjrsS6qRgkvd
The upload tag is computed as TAG_UPLOAD_CHUNK_MIN + upload_index, and the
assertion tag <= TAG_UPLOAD_CHUNK_MAX was failing when the number of uploads
exceeded 1000. This can happen with small chunk sizes (e.g., 1 byte) and
large writes, where each chunk needs multiple upload operations for
replication.
Increase TAG_UPLOAD_CHUNK_MAX from 2000 to 100000 to accommodate up to 99000
concurrent upload operations per write.
When responding to a full chunk download request (full == 1), the code
was writing uninitialized target_len to the response message instead of
the actual data length. This caused chunk-to-chunk downloads to fail,
preventing proper replication and blocking coverage of download-related
code paths.
Two changes to improve code coverage in deterministic simulation testing:
1. Add 3 chunk servers (was 1) to match the replication factor
- This enables testing of chunk replication and chunk-to-chunk
server communication code paths in chunk_server.c
- Covers: process_chunk_server_download_*, start_download,
download_targets_* functions
2. Fix Quakey bug where events targeting a closed descriptor caused
assertion failures
- Added remove_events_targeting_desc() to clean up DISCONNECT and
DATA events when a descriptor is freed
- Modified desc_free() to accept Sim pointer and call cleanup
- This was exposed by running multiple chunk servers which create
more connection activity
Initialize *gen to NO_GENERATION at the start of file_tree_read so that
error paths (FILETREE_BADPATH, FILETREE_NOENT, FILETREE_ISDIR) have a
well-defined value. Previously, when file_tree_read failed, gen was
uninitialized, causing the assertion in process_client_read to fire.
The assertion checking gen != NO_GENERATION before the error check was
removed since:
1. On error paths, gen is not used
2. On success paths, the assertion at line 482 already validates gen
Adds build_coverage.sh script that:
- Builds the simulation with GCC branch coverage instrumentation
- Supports --report flag to run simulation and generate lcov HTML report
- Uses 60 second timeout with SIGINT for graceful shutdown
Also adds SIGINT handler to simulation to exit cleanly and flush coverage data.
Implements a new special generation counter value (UINT64_MAX) that
allows clients to assert that a file should NOT exist when performing
operations.
Primary use case: Atomic "create-if-not-exists" operations
When combined with TOASTY_WRITE_CREATE_IF_MISSING flag:
- File doesn't exist → server creates file and writes (SUCCESS)
- File exists → gen_match fails → BADGEN error (prevents overwrite)
This enables race-condition-free file creation where you want to
create a NEW file or fail if it already exists.
Key changes:
- Added MISSING_FILE_GENERATION constant to file_tree.h
- Updated gen_match() to return false when checking existing entities
against MISSING_FILE_GENERATION
- Modified file_tree_delete_entity() to succeed (no-op) when file is
missing and MISSING_FILE_GENERATION is specified (idempotent delete)
- Modified file_tree_write() to validate against the new value
- Added documentation in metadata_server.c explaining how
MISSING_FILE_GENERATION works WITH CREATE_IF_MISSING
- Updated protocol documentation in PROTOCOL.txt
- Added comprehensive feature documentation in MISSING_FILE_GENERATION.md
Additional use cases:
- Idempotent delete operations (delete only if not already gone)
- Detecting unexpected file creation in validation/testing scenarios
The implementation is backward compatible and type-safe. Files are
never assigned MISSING_FILE_GENERATION as their actual generation value.
Adds support for truncating files after write operations, enabling proper
HTTP PUT semantics where the entire file content should be replaced.
Implementation:
- Added TOASTY_WRITE_TRUNCATE_AFTER flag (0x02) to ToastyFS API
- Updated file_tree_write() to accept truncate_after parameter:
* When true, sets file size to exactly offset+length
* Removes chunks beyond the new file size
* Marks removed chunks for garbage collection
- Metadata server extracts and passes truncate flag to file_tree_write()
- WAL replay passes false for truncate_after (already handled in original write)
- Web proxy PUT now uses both CREATE_IF_MISSING and TRUNCATE_AFTER flags
- Updated documentation in PROTOCOL.txt, DESIGN.txt, and web/DESIGN.txt
Benefits:
- Proper HTTP PUT semantics (replace entire file content)
- Efficient: single write operation truncates and updates file
- Works with both sync and async APIs
- Garbage collection automatically removes orphaned chunks
Adds support for automatically creating files when write operations target
non-existent files. This is essential for REST PUT operations which should
be able to create new resources.
Implementation:
- Added TOASTY_WRITE_CREATE_IF_MISSING flag to ToastyFS API header
- Updated toasty_write() and toasty_begin_write() to accept flags parameter
- Client sends write flags in MESSAGE_TYPE_WRITE message to metadata server
- Metadata server parses flags and handles file auto-creation atomically:
* When write fails with FILETREE_NOENT and flag is set
* Logs creation to WAL for crash consistency
* Creates file with default 4096-byte chunk size
* Retries write with newly created file's generation tag
- Updated web proxy PUT handler to use TOASTY_WRITE_CREATE_IF_MISSING
- Updated examples and simulation client for new API signature
Benefits:
- Works for both sync and async APIs
- Single round trip (atomic server-side operation)
- Prevents race conditions
- Proper WAL logging ensures crash consistency
cozis
Claude