The COMMIT_PUT message was built with request_id N, then request_id
was incremented to N+1 immediately after sending. When the server
replied with request_id N, the client compared it against N+1,
silently discarded the reply, and waited for the 5-second timeout.
The timeout retry (replay_request) then resent with N+1, which
matched — so every PUT paid an unnecessary 5-second penalty.
Move the request_id++ before building the CommitPutMessage, matching
the pattern already used by toastyfs_async_delete.
https://claude.ai/code/session_01EG3WZSH3ZeTdAbb5T42r2w
When no TCP connection exists, send_message_to_server() called
tcp_connect() with a NULL output parameter and returned without
sending the message. The message was silently dropped, and the only
recovery was the 5-second PRIMARY_DEATH_TIMEOUT_SEC retry — causing
the client to hang on every cold-connection operation.
tcp_connect() already supports returning the output ByteQueue so
callers can queue data to be flushed once the connection completes.
Use that mechanism: get the output queue from tcp_connect() and
write the message into it immediately, so it is sent as soon as
the TCP handshake finishes.
https://claude.ai/code/session_01EG3WZSH3ZeTdAbb5T42r2w
Four bugs caused data mismatches between PUT and GET operations:
1. chunk_store_exists() used file_open (O_CREAT) to check if a chunk
file exists, which created empty files as a side effect. Servers
would then read from these empty files and return uninitialized
heap memory as chunk data. Fixed by using file_exists (access())
instead.
2. Server's process_fetch_chunk checked `ret < 0` after
chunk_store_read, missing the EOF case (ret == 0) from empty
files. Changed to `ret <= 0`.
3. begin_transfers() == 0 was used as a completion check in both
STEP_FETCH_CHUNK and STEP_STORE_CHUNK, but returning 0 only means
"no new transfers started", not "all done". With concurrent chunk
transfers, this caused premature completion, leaving in-flight
transfers to hit UNREACHABLE. Fixed by using
all_chunk_transfers_completed() instead.
4. choose_store_locations_for_chunk() returned [0, 1] for ALL chunks
regardless of index, while GET fetches chunk i from servers
(i+j) % num_servers. This made some chunks unreachable. Fixed by
using (chunk_idx + j) % num_servers for store locations.
Also added SO_REUSEADDR to the server listen socket so restarting
the cluster doesn't fail with "Couldn't setup TCP listener", and
added a note on mock_access about the simulation mismatch.
https://claude.ai/code/session_01JniNxxDP4NbsDXNGmGNG7H
The client library had several issues preventing operations from completing
in the simulation:
- toastyfs_register_events discarded the computed timeout instead of
returning it to callers, so clients never woke up on schedule
- random_client_tick ignored the timeout parameter entirely
- deadline_to_timeout truncated sub-millisecond values to 0 instead of
rounding up, causing busy-wait loops
- The server didn't reset the heartbeat timer after completing a view
change, so followers would immediately timeout again
- The client's DELETE handler didn't handle META_RESULT_NOT_FOUND,
causing an assertion failure when deleting non-existent keys
- Most critically, messages sent before TCP connections were established
were silently dropped with no retry mechanism — added timeout-based
retry logic in toastyfs_process_events
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
Add logging infrastructure to client.c mirroring the server's node_log
pattern, with trace lines for: init, async put/get/delete, begin
transfers, replay requests, all received message types (store ack, fetch
chunk response, redirect, reply, get blob response), disconnects, and
unexpected messages.
Add result/operation trace lines to random_client.c and fix a bug where
the random client never started its first operation (the condition
`result.type != TOASTYFS_RESULT_VOID` was always false on the first
tick since no operation was pending).
https://claude.ai/code/session_0184gHjra7fsmSPZ4kppaGhC
- Add #warning to remind about replacing compute_chunk_hash with proper SHA256
- Define MAX_TRANSFERS and CEIL macros in config.h and use them
- Add TODO noting server selection formula is temporary
- Add NOTE about sticky error pattern in async functions
- Add TODO for INVALID_TIME error handling at all get_current_time call sites
- Add TODO about determining timeout based on pending operation status
https://claude.ai/code/session_01TkNavR4qyZcTuXMPmrtWUJ
- Add REPLICATION_FACTOR and CHUNK_SIZE constants to config.h
- Add proper error codes (REJECTED, FULL, NOT_FOUND, TRANSFER_FAILED, etc.) to toastyfs.h
- Implement compute_chunk_hash() for deterministic chunk hashing
- Implement replay_request() to resend operations after leader redirect
- Implement send_message_to_server_ex() for sending messages with trailing data
- Implement choose_store_locations_for_chunk() to pick REPLICATION_FACTOR servers
- Fix begin_transfers() to handle both StoreChunk (PUT) and FetchChunk (GET)
- Complete toastyfs_async_put(): chunk splitting, hashing, data copy, metadata setup
- Complete toastyfs_async_get()/delete(): key setup, step/time initialization
- Fix get_result(): use actual transfer sizes instead of fixed chunk_size, free transfer data
- Replace all xxx placeholders and TOASTYFS_ERROR_XXX with proper values
- Add chunk_sizes[] tracking and put_data lifetime management
- Implement random_client.c: rng(), choose_random_oper(), proper API call arguments
https://claude.ai/code/session_01TkNavR4qyZcTuXMPmrtWUJ
- Rename phase_time to step_time for consistency with Step enum
- Unify blob_size and file_size into single blob_size field
- Merge toastyfs_alloc into toastyfs_init, returning malloced ToastyFS*
- Replace (MessageHeader*)&msg casts with &msg.base
- Replace !resp.size with explicit resp.size == 0
https://claude.ai/code/session_018MRQTdAZoBCXJZgdP4U6V6
1. Fix client not reading all server addresses in READ response
The metadata server sends address info for ALL chunk servers holding
a chunk, but the client only read info for ONE server. This caused
message parsing to become misaligned for subsequent chunks.
Added a loop to iterate through all num_servers and read each
server's address list (using the first one, skipping the rest).
2. Fix tag value conflict between metadata and chunk server responses
TAG_RETRIEVE_METADATA_FOR_READ was 1, which conflicted with chunk
download range indices (0, 1, 2, 3...). When the second chunk
download response came with range_idx=1, it was incorrectly treated
as a metadata response.
Changed TAG values from positive to negative integers since range
indices are always non-negative.
Also updated the test to properly exit on success instead of assert(0).
https://claude.ai/code/session_01QYDQCNiUj1cjrsS6qRgkvd
The upload tag is computed as TAG_UPLOAD_CHUNK_MIN + upload_index, and the
assertion tag <= TAG_UPLOAD_CHUNK_MAX was failing when the number of uploads
exceeded 1000. This can happen with small chunk sizes (e.g., 1 byte) and
large writes, where each chunk needs multiple upload operations for
replication.
Increase TAG_UPLOAD_CHUNK_MAX from 2000 to 100000 to accommodate up to 99000
concurrent upload operations per write.
Implements a new special generation counter value (UINT64_MAX) that
allows clients to assert that a file should NOT exist when performing
operations.
Primary use case: Atomic "create-if-not-exists" operations
When combined with TOASTY_WRITE_CREATE_IF_MISSING flag:
- File doesn't exist → server creates file and writes (SUCCESS)
- File exists → gen_match fails → BADGEN error (prevents overwrite)
This enables race-condition-free file creation where you want to
create a NEW file or fail if it already exists.
Key changes:
- Added MISSING_FILE_GENERATION constant to file_tree.h
- Updated gen_match() to return false when checking existing entities
against MISSING_FILE_GENERATION
- Modified file_tree_delete_entity() to succeed (no-op) when file is
missing and MISSING_FILE_GENERATION is specified (idempotent delete)
- Modified file_tree_write() to validate against the new value
- Added documentation in metadata_server.c explaining how
MISSING_FILE_GENERATION works WITH CREATE_IF_MISSING
- Updated protocol documentation in PROTOCOL.txt
- Added comprehensive feature documentation in MISSING_FILE_GENERATION.md
Additional use cases:
- Idempotent delete operations (delete only if not already gone)
- Detecting unexpected file creation in validation/testing scenarios
The implementation is backward compatible and type-safe. Files are
never assigned MISSING_FILE_GENERATION as their actual generation value.
Adds support for automatically creating files when write operations target
non-existent files. This is essential for REST PUT operations which should
be able to create new resources.
Implementation:
- Added TOASTY_WRITE_CREATE_IF_MISSING flag to ToastyFS API header
- Updated toasty_write() and toasty_begin_write() to accept flags parameter
- Client sends write flags in MESSAGE_TYPE_WRITE message to metadata server
- Metadata server parses flags and handles file auto-creation atomically:
* When write fails with FILETREE_NOENT and flag is set
* Logs creation to WAL for crash consistency
* Creates file with default 4096-byte chunk size
* Retries write with newly created file's generation tag
- Updated web proxy PUT handler to use TOASTY_WRITE_CREATE_IF_MISSING
- Updated examples and simulation client for new API signature
Benefits:
- Works for both sync and async APIs
- Single round trip (atomic server-side operation)
- Prevents race conditions
- Proper WAL logging ensures crash consistency
Updated both client.c and metadata_server.c to match the protocol
specification in misc/PROTOCOL.txt:
Client-to-Server Messages:
- Added expect_gen field to DELETE, LIST, READ, and WRITE messages
- Removed old_hash and chunk_size from WRITE message chunks
Server-to-Client Responses:
- Added generation counter to CREATE_SUCCESS response
- Added generation counter to LIST_SUCCESS response
- Fixed LIST_SUCCESS field order (gen, truncated, item_count)
- Added generation counter to READ_SUCCESS response
- Added generation counter to WRITE_SUCCESS response
All changes compile successfully and pass valgrind memory checks.
Add functionality to determine the actual number of bytes read during
read operations, making it possible to detect when a read was truncated
because it went past the end of the file.
Changes:
- Modified file_tree_read() to calculate and return actual_bytes via
new output parameter
- Updated metadata server to send actual_bytes in READ_SUCCESS messages
- Added bytes_read field to ToastyResult structure
- Modified client to parse, store, and report actual bytes read
- Updated toasty_read() to return the actual number of bytes read
instead of always returning 0
- Fixed web server to use bytes_read field instead of non-existent
count field
This allows clients to distinguish between:
- Reading zeros because the file is sparse (has holes)
- Reading past the end of the file (truncated read)
cozis
Claude