1. Fix client not reading all server addresses in READ response
The metadata server sends address info for ALL chunk servers holding
a chunk, but the client only read info for ONE server. This caused
message parsing to become misaligned for subsequent chunks.
Added a loop to iterate through all num_servers and read each
server's address list (using the first one, skipping the rest).
2. Fix tag value conflict between metadata and chunk server responses
TAG_RETRIEVE_METADATA_FOR_READ was 1, which conflicted with chunk
download range indices (0, 1, 2, 3...). When the second chunk
download response came with range_idx=1, it was incorrectly treated
as a metadata response.
Changed TAG values from positive to negative integers since range
indices are always non-negative.
Also updated the test to properly exit on success instead of assert(0).
https://claude.ai/code/session_01QYDQCNiUj1cjrsS6qRgkvd
The upload tag is computed as TAG_UPLOAD_CHUNK_MIN + upload_index, and the
assertion tag <= TAG_UPLOAD_CHUNK_MAX was failing when the number of uploads
exceeded 1000. This can happen with small chunk sizes (e.g., 1 byte) and
large writes, where each chunk needs multiple upload operations for
replication.
Increase TAG_UPLOAD_CHUNK_MAX from 2000 to 100000 to accommodate up to 99000
concurrent upload operations per write.
When responding to a full chunk download request (full == 1), the code
was writing uninitialized target_len to the response message instead of
the actual data length. This caused chunk-to-chunk downloads to fail,
preventing proper replication and blocking coverage of download-related
code paths.
Two changes to improve code coverage in deterministic simulation testing:
1. Add 3 chunk servers (was 1) to match the replication factor
- This enables testing of chunk replication and chunk-to-chunk
server communication code paths in chunk_server.c
- Covers: process_chunk_server_download_*, start_download,
download_targets_* functions
2. Fix Quakey bug where events targeting a closed descriptor caused
assertion failures
- Added remove_events_targeting_desc() to clean up DISCONNECT and
DATA events when a descriptor is freed
- Modified desc_free() to accept Sim pointer and call cleanup
- This was exposed by running multiple chunk servers which create
more connection activity
Initialize *gen to NO_GENERATION at the start of file_tree_read so that
error paths (FILETREE_BADPATH, FILETREE_NOENT, FILETREE_ISDIR) have a
well-defined value. Previously, when file_tree_read failed, gen was
uninitialized, causing the assertion in process_client_read to fire.
The assertion checking gen != NO_GENERATION before the error check was
removed since:
1. On error paths, gen is not used
2. On success paths, the assertion at line 482 already validates gen
Adds build_coverage.sh script that:
- Builds the simulation with GCC branch coverage instrumentation
- Supports --report flag to run simulation and generate lcov HTML report
- Uses 60 second timeout with SIGINT for graceful shutdown
Also adds SIGINT handler to simulation to exit cleanly and flush coverage data.
Implements a new special generation counter value (UINT64_MAX) that
allows clients to assert that a file should NOT exist when performing
operations.
Primary use case: Atomic "create-if-not-exists" operations
When combined with TOASTY_WRITE_CREATE_IF_MISSING flag:
- File doesn't exist → server creates file and writes (SUCCESS)
- File exists → gen_match fails → BADGEN error (prevents overwrite)
This enables race-condition-free file creation where you want to
create a NEW file or fail if it already exists.
Key changes:
- Added MISSING_FILE_GENERATION constant to file_tree.h
- Updated gen_match() to return false when checking existing entities
against MISSING_FILE_GENERATION
- Modified file_tree_delete_entity() to succeed (no-op) when file is
missing and MISSING_FILE_GENERATION is specified (idempotent delete)
- Modified file_tree_write() to validate against the new value
- Added documentation in metadata_server.c explaining how
MISSING_FILE_GENERATION works WITH CREATE_IF_MISSING
- Updated protocol documentation in PROTOCOL.txt
- Added comprehensive feature documentation in MISSING_FILE_GENERATION.md
Additional use cases:
- Idempotent delete operations (delete only if not already gone)
- Detecting unexpected file creation in validation/testing scenarios
The implementation is backward compatible and type-safe. Files are
never assigned MISSING_FILE_GENERATION as their actual generation value.
Adds support for truncating files after write operations, enabling proper
HTTP PUT semantics where the entire file content should be replaced.
Implementation:
- Added TOASTY_WRITE_TRUNCATE_AFTER flag (0x02) to ToastyFS API
- Updated file_tree_write() to accept truncate_after parameter:
* When true, sets file size to exactly offset+length
* Removes chunks beyond the new file size
* Marks removed chunks for garbage collection
- Metadata server extracts and passes truncate flag to file_tree_write()
- WAL replay passes false for truncate_after (already handled in original write)
- Web proxy PUT now uses both CREATE_IF_MISSING and TRUNCATE_AFTER flags
- Updated documentation in PROTOCOL.txt, DESIGN.txt, and web/DESIGN.txt
Benefits:
- Proper HTTP PUT semantics (replace entire file content)
- Efficient: single write operation truncates and updates file
- Works with both sync and async APIs
- Garbage collection automatically removes orphaned chunks
Adds support for automatically creating files when write operations target
non-existent files. This is essential for REST PUT operations which should
be able to create new resources.
Implementation:
- Added TOASTY_WRITE_CREATE_IF_MISSING flag to ToastyFS API header
- Updated toasty_write() and toasty_begin_write() to accept flags parameter
- Client sends write flags in MESSAGE_TYPE_WRITE message to metadata server
- Metadata server parses flags and handles file auto-creation atomically:
* When write fails with FILETREE_NOENT and flag is set
* Logs creation to WAL for crash consistency
* Creates file with default 4096-byte chunk size
* Retries write with newly created file's generation tag
- Updated web proxy PUT handler to use TOASTY_WRITE_CREATE_IF_MISSING
- Updated examples and simulation client for new API signature
Benefits:
- Works for both sync and async APIs
- Single round trip (atomic server-side operation)
- Prevents race conditions
- Proper WAL logging ensures crash consistency
Per PROTOCOL.txt specification, WRITE operations must provide a valid
generation counter and cannot use expect_gen=0 (unlike other operations).
This ensures the client has retrieved the file's metadata and knows the
correct chunk size before writing.
Changes:
- Added FILETREE_BADGEN error code (-8)
- Modified file_tree_write() to reject expect_gen=0
- Added error message for FILETREE_BADGEN
Updated both client.c and metadata_server.c to match the protocol
specification in misc/PROTOCOL.txt:
Client-to-Server Messages:
- Added expect_gen field to DELETE, LIST, READ, and WRITE messages
- Removed old_hash and chunk_size from WRITE message chunks
Server-to-Client Responses:
- Added generation counter to CREATE_SUCCESS response
- Added generation counter to LIST_SUCCESS response
- Fixed LIST_SUCCESS field order (gen, truncated, item_count)
- Added generation counter to READ_SUCCESS response
- Added generation counter to WRITE_SUCCESS response
All changes compile successfully and pass valgrind memory checks.
Previously, file_size was calculated as num_chunks * chunk_size, which
incorrectly treated the file size as the full capacity of all allocated
chunks. The actual file size should be the offset of the last byte
written plus 1.
Changes:
- Added file_size field to File structure to track actual file extent
- Initialize file_size to 0 when creating new files
- Update file_size in file_tree_write() based on write offset + length
- Modified file_tree_read() to use file_size instead of num_chunks * chunk_size
- Updated serialization/deserialization to handle file_size field
This ensures that actual_bytes calculations correctly reflect the true
file size, not just the allocated chunk capacity.
Add functionality to determine the actual number of bytes read during
read operations, making it possible to detect when a read was truncated
because it went past the end of the file.
Changes:
- Modified file_tree_read() to calculate and return actual_bytes via
new output parameter
- Updated metadata server to send actual_bytes in READ_SUCCESS messages
- Added bytes_read field to ToastyResult structure
- Modified client to parse, store, and report actual bytes read
- Updated toasty_read() to return the actual number of bytes read
instead of always returning 0
- Fixed web server to use bytes_read field instead of non-existent
count field
This allows clients to distinguish between:
- Reading zeros because the file is sparse (has holes)
- Reading past the end of the file (truncated read)
- Added mock_lseek declaration and implementation for Linux
- Added mock_SetFilePointer declaration and implementation for Windows
- Added sys_lseek and sys_SetFilePointer macros for both BUILD_TEST and production modes
- Mocks follow existing pattern: validate descriptors, check type, forward to real functions
- Both mocks properly handle error cases and set errno/SetLastError
cozis
Claude